Blog

Peer Review, Imperfect Feedback

June 26, 2023

I’ve been pretty critical of peer review in the past, arguing that it doesn’t accomplish much, contributes to status quo bias, etc. But a few recent experiences remind me of the value that peer review provides: in today’s scientific culture, peer review is essentially the only time that scientists get honest and unbiased feedback on their work.

How can this be true? In experimental science, scientists typically work alongside other students and postdocs under the supervision of a professor. This body of people forms a lab, also known as a research group, and it’s to these people that you present most frequently. Your lab generally knows the techniques and methods that you employ very well: so if you’ve misinterpreted a piece of data or designed an experiment poorly, group meeting is a great place to get feedback.

But a lab is also biased in certain ways. People are attracted to a lab because they think the science is exciting and shows promise, and so they’re likely to be credulous about positive results. Certain labs also develop beliefs or dogmas about how to conduct science: the best ways to perform a mechanistic study, or the most useful reaction conditions. To some extent, every lab is a paradigm unto itself. This means that paradigm-shifting criticism is hard to find among one’s coworkers, even if it’s common in the outside world.

Here are some examples of controversial-in-the-field statements that are unlikely to be controversial within given labs:

Palladium-based catalysis is going to become obsolete due to the scarcity of Pd and must be replaced. Some labs build their research program around this; others think Pd will always be relevant.
Water is the greenest possible solvent. Some scientists believe this wholeheartedly, while others think it’s stupid (for instance, a 2021 review states that “to meet regulations concerning the discharge of waste water into rivers and other natural waters… [water] often requires very extensive treatment prior to discharge, making the use of water as a reaction medium much less attractive.”)
More broadly, the belief that machine learning and generative AI are the future of chemical discovery: while many computational chemists believe this, plenty of experimentalists (even young ones) are pretty skeptical.

In each of these cases, it’s unlikely that criticism along these lines is available internally: people who’ve chosen to do their PhDs studying ML in chemistry aren’t likely to criticize your paper for overemphasizing the importance of ML in chemistry!

More generally, internal criticism works best when a lab serves as a shared repository of expertise, i.e. when everyone in the lab has roughly the same skillset. Some labs focus instead on a single overarching goal and employ many different tools to get to that point: a given chemical biology group might have a synthetic chemist, a MS specialist, a genomics guru, a mechanistic enzymologist, and someone specializing in cell culture. If this is the case, your techniques are opaque to your coworkers: what advice can someone who does cell culture give about improving Q-TOF signal-to-noise?

Ideally, one’s professor is well-versed enough in each of the techniques employed that he or she can dispense criticism as needed. But professors are often busy, aren’t always operational experts at each of the techniques they oversee, and suffer from the same viewpoint biases that their students do (perhaps even more so).

So, it’s important to solicit feedback from external sources. Unfortunately, at least in my experience most external feedback is too positive: “great talk,” “nice job,” etc. Our scientific culture tries so hard to be supportive that I almost never get any meaningful criticism from people outside my group, either publicly or privately. (Ideally one’s committee would help, but I never really got to present research results to my committee, and this doesn’t help postdocs anyhow.)

Peer review, then, serves as the last bastion against low-quality science: reviewers are outside the lab, have no incentive to be nice, and are tasked specifically with poking holes in your argument or pointing out extra experiments that would improve it. Peer review has improved each one of my papers, and I’m grateful for it.¹

What’s a little sad is that the excellent feedback that reviewers give only comes at the bitter end of a project, which for me has often meant that the results are more than a year old and my collaborators have moved on. Much more useful would be critical feedback delivered early on in a project, when my own thinking is more flexible and the barrier to running additional experiments is lower. And more useful still would be high-quality criticism available at every step of the project, given not anonymously but by people whom you can talk to and learn from.

What might this practically look like?

A culture of “red teaming,” where students are incentivized to find flaws in others’ projects in somewhat adversarial ways. This would need to be done within a supportive and collegial atmosphere, lest it degenerate into bullying: red teaming need not be red in tooth and claw.
Similarly, PIs could invite other professors to come to group meetings and (constructively) criticize the projects, particularly professors from adjacent subfields who might have different perspectives.
Poster presentations or talks (at conferences), although often used to present finished projects, can also be used to present unfinished work. I presented some unfinished work at TPOC a few years ago, and got really helpful suggestions from Ken Houk and Dean Tantillo that fundamentally changed how we approached the rest of the project. Maybe this is something that we should encourage more, although finished projects will probably always look more impressive than unfinished projects.
Decentralized peer review solutions like Seeds of Science or PubPeer might also help here, but my sense is that it’s unlikely that qualified experts will just spend their time investigating something that they found online; they need to be solicited somehow.

I don’t know what the right solution looks like here: the burden of peer review is already substantial, and I don’t mean to suggest that this work ought to be arbitrarily multiplied for free. But I do worry that eliminating peer review, absent other changes, would simply mean that one of the only meaningful chances to get unfiltered feedback on one’s science would be eliminated, and that this would be bad.

Thanks to Croix Laconsay and Lucas Karas for helpful feedback on this piece.

Footnotes

It's also true that the threat of peer review increases paper quality. I agree with Singleton that this is important today, but am less convinced that this is necessary from an institutional design perspective: if peer review didn't exist, I think some other system of norms or regulations would spring forth to protect good-quality science. See the "Anarchic Preprint Lake" discussion in my piece on journals. (h/t Croix Laconsay for raising this point)

A Farewell to Lab

June 22, 2023

I first encountered organic chemistry on Wikipedia, my freshman year of high school. The complexity and arcanity of the field instantly attracted me: here was something interesting that I didn’t know about and which didn’t require years of mathematical training to approach (unlike most of physics).

I soon started reading about organic chemistry more and more, albeit with no rhyme or reason to my study. I didn’t know what the good textbooks were, what order to study things in or which concepts ought to be understood in depth before progressing further. Organic chemistry was just a “glass bead game” to me, an art of symbols devoid of any real-world representations. But enthusiasm can sometimes suffice where wisdom is lacking.

With the support of a teacher and a half-dozen friends who also wanted to learn more organic chemistry, we started a little independent study. We met in a closet and read a textbook, worked through the problems, and our teacher wrote us tests. We eventually managed to get through all of Paula Bruice, although various misconceptions (and mispronunciations) stayed with me until I took Movassaghi’s course at MIT.

Here’s a picture of some work from high school, which does not in fact make very much sense. (I think this is from the Evans problem set library.)

But in high school we had no lab, and thus no practical knowledge. We split into two groups and tried to come up with experiments for ourselves, but the results were dismal. Here’s the procedure (copied verbatim) for the one and only reaction we ran, synthesis of nitrobenzene via nitration of benzene, which was to be the first step in a multistep synthesis of Kevlar:

In a 250 mL beaker, dissolve benzene in a solution of concentrated H₂SO₄ of twice its volume. Use an ice-salt bath to bring this solution to 0 °C or below, and use a Pasteur pipette to slowly add a 1:1 solution of HNO₃ and H₂SO₄ (be sure to keep the solution below 15 degrees Celsius at all times, as the reaction is strongly exothermic). Once all the solution has been added, warm it to room temperature and allow it to sit for 15 minutes.

Pour the solution over 50 g of crushed ice in a 250 mL beaker. Once the ice has melted, isolate the product via vacuum filtration via a Buchner funnel and rinse it twice with water and twice with methanol. Recrystallize in a solution of methanol.

The astute observer will notice that there aren’t very many details here. How much benzene? How much nitric acid? We didn’t have the glassware mentioned above—neither a beaker nor a Buchner funnel. And, perhaps most damningly, the procedure calls for isolation by filtration, challenging since nitrobenzene is a liquid at room temperature.

Despite these problems, we successfully ran this reaction (open to air, not in a fume hood), and obtained the product. I vividly recall the yellow bubbles of nitrobenzene floating to the top of the vial, and the smell of cherries that filled the room, a smell that returns to me in Proustian fashion from time to time when using certain reagents. We didn’t have a separatory funnel (or we didn’t know how to use it if we did), so we fished some of the nitrobenzene out with a Pasteur pipette and threw the rest away.

A little bit of knowledge would have served us well, as would have gloves and a fume hood. Here’s Wikipedia:

Prolonged exposure [to nitrobenzene] may cause serious damage to the central nervous system, impair vision, cause liver or kidney damage, anemia and lung irritation. Inhalation of vapors may induce headache, nausea, fatigue, dizziness, cyanosis, weakness in the arms and legs, and in rare cases may be fatal…

Nitrobenzene is considered a likely human carcinogen by the United States Environmental Protection Agency, and is classified by the IARC as a Group 2B carcinogen which is "possibly carcinogenic to humans".

Indeed, a coworker of mine would later be sent to the hospital after spilling nitrobenzene on herself. Surprisingly, my group’s experiment still ended up being the safer one—the lab portion of our course was disbanded the following day after my classmates caused an explosion with thionyl chloride.

I was fortunate enough to land a summer research internship in the Sessler lab at UT the following summer, and I started studying chemistry in earnest: column chromatography, Anslyn/Dougherty, NMR spectroscopy, and all the rest. I remember the first reaction I ran at UT: retro-Friedel–Crafts dealkylation of a calix[4]arene, using about 10 g each of phenol and aluminum(III) chloride. The brutal physicality of lab work was a nice contrast to the gnosticism of software (where I’d worked previously), and I was hooked.

I’ll fast-forward through the more recent parts of my chemical career: I went to MIT and joined the Imperiali lab to work on essentially a medicinal chemistry project: hit-to-lead optimization, featuring lots of de novo heterocycle synthesis. I got to cook reactions in molten urea, quench 500 mL of phosphorus oxychloride at a time, and even design new routes (with a little oversight from my postdoc). It was awesome.

My hood in the Imperiali Group. Although this hood appears to have a Schlenk line, do not be fooled—it’s purely decorative.

After three semesters, I got tired of my cross couplings mysteriously failing and joined the Buchwald lab, where I learned how to do chemistry more carefully: handling air-sensitive materials with a Schlenk line or in the glovebox, not “as fast as possible.” My tenure in the Buchwald lab also introduced me to the importance of computations, which became a key part of my doctoral work, particularly when we were sent home in March of my first year. COVID gave me the opportunity to pursue some software engineering projects that I wouldn’t have had time to work on otherwise (like cctk and presto), and simulation started to take up more and more of my time and intellectual bandwidth.

A large-scale biaryl synthesis proceeding through a benzyne intermediate, from my time in the Buchwald Group.

One of my first Gaussian jobs ever. I didn't get cluster access for a little while, so I used to run Gaussian on the printer computers in my dorm's basement and just hope that nobody needed the computer overnight.

I defended my dissertation on June 5th, and cleaned out my hood last week. For the foreseeable future, I’ll be a purely computational chemist—computation is advancing quickly, and I think that’s where I have the most to offer the field right now. But I’ll miss the sights and sounds of the lab. There’s a satisfaction to making a new molecule and holding the final product in your hands: the knowledge that you’ve reshaped this little corner of reality through your own actions, and that this particular arrangement of matter has never existed before.

And simulation is, at the end of the day, only useful insofar as it helps us make real molecules. There may be people who wish to model reactions purely for the sake of modeling them, but I am not one of them. What drew me to simulation initially, and what still attracts me to the field today, is its potential to help experimental scientists do their work faster and better. It was easy for me to ensure that this was true when I was both doing the experiments and running the simulations; my incentives were aligned properly. It will be harder in the future.

There’s a seductive appeal in leaving lab work behind altogether, too, and one that’s dangerous. Any experimentalist who’s worked with a computational collaborator knows that nothing ever works precisely as modeled. There are untold depths of chemical behavior still inaccessible to the idealized world of simulation, and it’s all too easy for computational chemists just to look the other way. Life is easier when you don’t have to deal with sludgy workups or poorly soluble intermediates, but they don’t go away just because you can’t model them by DFT—reality has a way of keeping us honest that simulations frequently lack.

So, although I bid lab farewell at this point in my career, it’s a bittersweet parting. I hope to return someday; only time will tell.

Thanks to Jacob Thackston for reading a draft of this piece.

Crystallization in Microgravity

June 16, 2023

Not Boring recently published a panegyric about Varda, a startup that’s trying to create “space factories making drugs in orbit.” When I first read this description, alarm bells went off in my head—why would anyone try to make drugs in space? Nevertheless, there’s more to this idea than I initially thought. In this piece, I want to dig a little deeper into the chemistry behind Varda, and discuss some potential advantages and challenges of the approach they’re exploring.

Much of my confusion was quickly resolved by realizing that Varda is not actually “making drugs in orbit,” or not in the way that an organic chemist would interpret that sentence. Varda’s proposal is actually much more specific: they aim to crystallize active pharmaceutical ingredients (APIs, i.e. finished drug molecules) in microgravity, allowing them to access crystal forms and particle size distributions which can’t be made under terrestrial conditions. To quote from their website:

Varda’s microgravity platform is grounded in decades of proven research conducted on NASA’s space stations. By eliminating factors such as natural convection and sedimentation, processing in a microgravity environment provides a unique path to formulating small molecules and biologics that traditional manufacturing processes cannot address. The resulting tunable particle size distributions, more ordered crystals, and new forms can lead to improved bioavailability, extended shelf-life, new intellectual property, and new routes of administration.

Crystallization is an excellent target for a new and expensive manufacturing process because it’s at once very important and very hard to control. The goal of crystallization is to grow crystals of a given compound, fitting the component molecules together into a perfect lattice that excludes impurities and can be easily handled. To the best of my knowledge, almost every small-molecule API is crystallized at one stage or another; it’s the best way to ensure that the material is extremely pure.

(Crystallizing proteins for administration is less common, since proteins are really difficult to crystallize, but it’s not unheard of—insulin is often administered subcutaneously as a microcrystalline suspension, which allows higher concentrations to be accessed without excessive viscosity.)

But crystallization is also something we can’t really control. We can’t physically put molecules into a lattice or force them to adopt ordered configurations; all we can do is dissolve them in some mixture, tweak the conditions a little bit, and hope that crystals form. Thus, finding good crystallization conditions basically amounts to randomly screening solvents and additives, leaving the solutions for a long time, and checking to see if crystals grow. In the words of Derek Lowe:

I'd like to open up the floor for nominations for the Blackest Art in All of Chemistry. And my candidate is a strong, strong contender: crystallization. When you go into a protein crystallography lab and see stack after stack after stack of plastic trays, each containing scores of different little wells, each with a slight variation on the conditions, you realize that you're looking at something that we just don't understand very well.

Why does microgravity matter for crystallization? Not Boring says that crystallization occurs “at the mesoscopic scale, the length scale on which objects are larger than nanoscale (on the order of atoms and molecules) but still small enough that quantum mechanical or other non-trivial ‘microscopic’ behavior becomes apparent.” I found this answer a little confusing—doesn’t crystallization begin on the nanoscale and end on the macroscopic scale?

Clearer to me was the explanation from a 2001 review by Kundrot and co-workers:

In zero gravity, a crystal is subject to Brownian motion as on the ground, but unlike the ground case, there is no acceleration inducing it to sediment [fall out of solution]. A growing crystal in zero gravity will move very little with respect to the surrounding fluid. Moreover, as growth units leave solution and are added to the crystal, a region of solution depleted in protein is formed. Usually this solution has a lower density than the bulk solution and will convect upward in a 1g field as seen by Schlerien photography (Figure 1). In zero gravity, the bouyant [sic] force is eliminated and no bouyancy-driven convection occurs. Because the positions of the crystal and its depletion zone are stable, the crystal can grow under conditions where its growing surface is in contact with a solution that is only slightly supersaturated. In contrast, the sedimentation and convection that occur under 1g place the growing crystal surface in contact with bulk solution that is typically several times supersaturated. Lower supersaturation at the growing crystal surface allows more high-energy misincorporated growth units to disassociate from the crystal before becoming trapped in the crystal by the addition of other growth units…

In short, promotion of a stable depletion zone in microgravity is postulated to provide a better ordered crystal lattice and benefit the crystal growth process.

To summarize, microgravity serves to immobilize the crystal with respect to the surrounding solution, preventing convection or sedimentation from bringing highly concentrated solutions into contact with the crystal. This slows crystal growth, which might sound bad but is actually really good: in general, the slower a crystal grows, the higher its purity. (See also this 2015 article for further discussion.)

What practical impact does this have? In most cases, crystals grown in space are better than their terrestrial congeners by a variety of metrics: larger, structurally better, and more uniform. To quote from the Not Boring piece:

Doing crystallization in space is like adding a gravity knob to your instrument—it opens up regions of process design space that would otherwise be inaccessible. Importantly, after the crystallization occurs in space, the drug retains its solid state upon re-entry. Manufacture in space; use on earth.

This is why pharma is going to space to experiment with a wide range of medicines. Formulations made in microgravity could open the door to improvements in drug shelf life, bioavailability, IP expansion, and even better approaches to drug delivery…

To date, there has been a major disconnect between microgravity research and manufacturing. While it’s been possible to hitch a ride to the ISS and collaborate with NASA on PCG experiments, there is no existing commercial offering to actually manufacture drugs in space. Merck used their research results on Keytruda® crystallization to tinker with their terrestrial approaches to formulation. What if they could actually just manufacture the crystals they discovered in microgravity at commercial scale?

This is Varda’s mission—to make widespread research and manufacturing in microgravity a reality.

One concern I have is that to date, the vast majority of space-based crystallization has been aimed at structural biology (elucidating the structure of a protein via crystallography), which only takes one crystal, one time. What Varda is aiming to do is preparative crystallography: crystallizing proteins and small molecules to isolate large quantities of them. Both processes obviously involve growing crystals, but otherwise they’re pretty different: in structural biology, all you care about is isolating a single large and very pure crystal, while uniformity and reproducibility are paramount in preparative crystallography.

There’s some precedent for preparative protein crystallization in microgravity: a 1996 Schering-Plough paper studied crystallization of zinc interferon-α2b on the Space Shuttle. The results are excellent: over 95% of the protein crystallized, and the resulting suspension showed good stability and improved serum half-life in Cynomolgus monkeys (ref, ref). The difference in crystal quality is huge:

Space-grown crystals are much, much larger than crystals grown on Earth.

More recently, scientists from Merck found that crystallization of pembrolizumab (Keytruda) in microgravity reproducibly formed a monomodal particle size distribution, as opposed to the bimodal particle size distribution formed under conventional conditions (ref), although the crystals didn’t seem any larger:

Comparison of pembrolizumab crystal size distribution under different conditions.

Of course, until recently access to space was very limited and there wasn’t much reason to study preparative protein crystallography in microgravity, so the lack of studies is hardly surprising. For all the reasons discussed above, it seems very likely that preparative crystallography will be generally better in microgravity, and that the resulting crystals will be more homogenous and more pure than the crystals you could grow on Earth. But it’s not 100% certain, and that’s something Varda will have to establish. The word “can” is doing a lot of work in this text from their website:

By eliminating factors such as natural convection and sedimentation, processing in a microgravity environment provides a unique path to formulating small molecules and biologics that traditional manufacturing processes cannot address. The resulting tunable particle size distributions, more ordered crystals, and new forms can lead to improved bioavailability, extended shelf-life, new intellectual property, and new routes of administration (emphasis added)

Another concern is that little work (that I’m aware of) has been done on small molecule crystallization in microgravity—the very task Varda intends to start with. Small molecules, in general, are much easier to crystallize than proteins, and there are more parameters for the experimental scientist to tune. While there are certainly cases where small molecule crystals can display unexpected or problematic behaviors (like the famous case of ritonavir, the molecule Varda is investigating first), in general crystallization of small molecules seems like an easier problem, and one for which there are better state-of-the-art workarounds.

The most likely failure mode to me, though, is just that microgravity crystallization is better than crystallization on Earth, but not that much better. Shooting APIs into space, waiting for them to crystallize, and then launching them back to Earth is going to be really expensive, and Varda will have to demonstrate extraordinary results to justify the added hassle—particularly for a technique that they hope to make a key part of the pharmaceutical manufacturing process. Talk about supply chain risk! (Is this all going to be GMP?)

And it’s worth pointing out a fairly obvious consideration too: what Varda is proposing to do in space is only one part of a vast, multi-step operation. No synthesis will take place in space, so all the manufacturing of either proteins or small molecules will take place on Earth. The purified products will then be launched into space, crystallized, and then the crystal-containing solution will undergo reentry and then be formulated into whatever final form the drug needs to be administered in.

So, although “there are a number of small molecules that fetch hundreds of thousands or millions of dollars per kg” (Not Boring), Varda can address only a small—albeit important—part of the manufacturing process. I doubt this is likely to change. On average, it takes 100–1000 kg of raw materials to manufacture 1 kg of a small molecule drug (ref), so shipping everything to orbit would massively raise costs, without any obvious advantages that I can think of. The TAM for Varda might be large enough to break even, but it’s not going to replace conventional pharmaceutical manufacturing anytime soon.

Varda’s pitch is perfect for venture capital: ambitious, risky, and potentially game-changing if it succeeds. And I wish them luck in their quest, since new and better ways to approach formulation would be awesome. But I can’t shake the nagging doubt that they’re so excited about the image of space-based manufacturing that they’re trying to invent a problem that their aerospace engineers have a solution for. We’ll find out soon enough if they’ve succeeded.

Blog

Peer Review, Imperfect Feedback

Footnotes

A Farewell to Lab

A Year In The Blog

Crystallization in Microgravity

Seven Degrees of Screening for Generality