The Multimodal Universe (MMU) pools together 80TB+ of data from over 30 astronomical surveys into one place. Crossmatching (linking observations of the same object across surveys) is its killer feature, but until now it required downloading hefty chunks of data to local disk. We got tired of needing a cluster just to run a crossmatch, so we gathered in the UniverseTBD and Hugging Science Discord servers to fix that. We've converted the MMU to the parquet-based HATS format so that you can use the LSDB and Hugging Face ecosystems to crossmatch from a laptop. The datasets are here https://huggingface.co/collections/UniverseTBD/multimodal-un.... No bulk downloads are necessary, and 4GB of RAM is enough even at Gaia scale.
We investigate the potential constraints on LLM scaling posed by the availability of public human-generated text data. We forecast the growing demand for training data based on current trends and estimate the total stock of public human text data. Our findings indicate that if current LLM development trends continue, models will be trained on datasets roughly equal in size to the available stock of public human text data between 2026 and 2032, or slightly earlier if models are overtrained. We explore how progress in language modeling can continue when human-generated text datasets cannot be scaled any further. We argue that synthetic data generation, transfer learning from data-rich domains, and data efficiency improvements might support further progress.
This work presents AstroPT, an autoregressive pretrained transformer developed with astronomical use-cases in mind. The AstroPT models presented here have been pretrained on 8.6 million 512 × 512 pixel grz-band galaxy postage stamp observations from the DESI Legacy Survey DR8. We train a selection of foundation models of increasing size from 1 million to 2.1 billion parameters, and find that AstroPT follows a similar saturating log-log scaling law to textual models. We also find that the models' performances on downstream tasks as measured by linear probing improves with model size up to the model parameter saturation point. We believe that collaborative community development paves the best route towards realising an open source `Large Observation Model' -- a model trained on data taken from the observational sciences at the scale seen in natural language processing. To this end, we release the source code, weights, and dataset for AstroPT under the MIT license, and invite potential collaborators to join us in collectively building and researching these models.
If you are interested in this also check out EarthPT, which is also a time series decoding transformer (and has the code and weights released under the MIT licence): https://arxiv.org/abs/2309.07207
Wanted to share the code release of EarthPT, a model that predicts future satellite observations in a zero shot setting! I'm the first author so please shoot any questions you have at me.
EarthPT is a 700 million parameter decoding transformer foundation model trained in an autoregressive self-supervised manner and developed specifically with EO use-cases in mind. EarthPT can accurately predict future satellite observations across the 400-2300 nm range well into the future (we found six months!).
The embeddings learnt by EarthPT hold semantically meaningful information and could be exploited for downstream tasks such as highly granular, dynamic land use classification.
The coolest takeaway for me is that EO data provides us with -- in theory -- quadrillions of training tokens. Therefore, if we assume that EarthPT follows neural scaling laws akin to those derived for Large Language Models (LLMs), there is currently no data-imposed limit to scaling EarthPT and other similar ‘Large Observation Models.’(!)
Please give it a read! It begins from first principles (Rosenblatts perceptron!) and builds from there so you might find it more general than you expect.
I wanted it to reach a more general audience, as the review is very general in itself (but maybe the original title does not reflect this as I thought)! There are alternating sections concentrating on the astronomy and the deep learning sides.
Author here! We explore the past, present, and future of deep learning in astronomy. We predict that GPT-like foundation models will make a huge impact on the field, and that astronomy is ideally placed to supercharge open source large language modelling (Section 9).
My favourite excerpt, where we propose foundation model-powered scientists:
Autonomous agents are no longer science fiction; task-driven autonomous agents powered by the simulacra of a foundation model are capable of solving very general tasks when given only a high-level prompt by a human operator [305,306]. One could therefore imagine a semi-automated research pipeline, where an autonomous agent with astronomical knowledge is given access to a set of astronomical data through an API. The agent would be prompted with a high-level research goal (such as ‘find something interesting and surprising within this dataset’), and would then take steps to achieve this task. These steps could include querying research papers for a literature review, searching a large multi-modal astronomical dataset to find data that supports a theory, evoking and discussing its findings with additional simulacra, or spinning up simulations to test a hypothesis [307]. While the agent operates in the background, the human researcher would be able to provide high-level interpretation of the results, and would be a steady hand providing guidance and refinement of a more general research direction. In this way, an astronomical foundation model would provide the tools to make all astronomers the principal investigator of their own powerful ‘AI lab’
I wrote a literature review on applying neural networks to astronomical problems -- I found that using applications really helped to iron out what is going on in the networks! Here's the link to the review https://arxiv.org/abs/2211.03796