![]() If you have a static webapp, you can power it with data that is kept evergreen by Flat (we built a simple example using GitHub pages). With codespaces, you can author Flat data workflows and develop applications that use that data without ever leaving your browser, and without that data ever reaching your machine. It makes this pattern dead simple for developers and accessible to other audiences that are already using GitHub to share and work with data, like journalists and scientists. ![]() Just evergreen data, right in your repo.įlat stands squarely on the shoulders of the git scraping pattern first articulated by Simon Willison. No new mental model to learn and incorporate. No dependencies, libraries, or package managers. That's it! No complicated job dependency graphs or orchestrators. Flat workflows usually run on a periodic timer, but can be triggered by a variety of stimuli, like changes to your code, or manual triggers. The resulting data is committed to your repository if the new data is different, with a commit message summarizing the changes. Each Flat workflow fetches the data you specify, and optionally executes a postprocessing script on the fetched data. It runs on GitHub Actions, so there's no infrastructure to provision and monitor. Surgery on data also requires specialized tools that offer unusual capabilities or guarantee certain behaviors! However, most medicine is not surgery, and we do ourselves a disservice if we accept that sort of complexity into every situation involving data.įlat Data aims to simplify everyday data acquisition and cleanup tasks. Surgeons have trays full of weird tweezer-like things whose shapes differ but whose ultimate purpose is the same: grabbing hold of stuff that might be difficult to grab with our fingers. It's easy to get dazzled by the complexity and diversity of data tooling, but we're not the first profession to invent hyper-specialized implements. Data architectures vary wildly, so these solutions have a wide range of ambition (what they attempt to do) and complexity (what mess they attempt to conceal.) This is in contrast to the application/compute space, where pithy, prescriptive manifestos like 12 factor offer a great lens through which to think: "Do it this way, and thou shalt scale." There is no equivalent for data because there are so many different approaches one can only nail so many theses to the church doors before running out of room to describe best practices. ![]() If I don't have the data, then either I alter my application logic to work with data at a distance, or I figure out how to bring a working set to my local environment.Īs a result, there's an entire industry of tools that accomplish the chore of getting data to the right place, in the right format, at the right time. If I don't have the data, then it doesn't matter if the data is cleaned, filtered, and sorted. If I have the data, I can load it, and get to work. Locality of data isn't some abstract concept when you're trying to build things on top of that data - it's the leading term of developer experiences. Beyond the pain of actually doing this work, it has become impossible to talk about this stuff without abusing metaphors beyond their safe design limits.Īs a developer, all those bits occupying the proverbial lake/warehouse/refinery are as immediately useful as a grape seed is to a winery. Directing those data tributaries into undifferentiated data lakes so that we may pose different queries to the data someday. ![]() Strategies for contending with a deluge of events from chatty devices. Techniques for sampling and transforming data while it moves instead of operating on it after it has come to rest. Distributed data systems that slip the surly bonds of any one machine to touch the face of scale. There’s a certain kind of language that dominates the discourse on data today. Btc-price-postprocessed.json Why Flat Data?
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |