Uber has introduced a new distributed computing library for computer clusters: Fiber. Fiber is Python-based and allows programmers to program a cluster of computers much in the way they would typically program a laptop or desktop. Uber had originally developed Fiber to program large scale computation projects like POET.
In its decision to create and distribute Fiber, Uber found that current technology used to run large scale distributing computing jobs is currently limited due to the gap between local code and cluster production, lack of dynamic scaling, improper error handling, and a high cost of learning. Uber believes Fiber fills these gaps. On the Fiber site, the Uber team states:
"The new Fiber platform addresses each of these issues explicitly, potentially opening up seamless large-scale distributed computing to a much wider population of users."
In its mission to solve these problems, Fiber was specifically designed to be easy to use, easy to learn, fast in performance, with reliable computation. But, perhaps most convincingly, Fiber is run the same way a normal application is run on a computer cluster.
Fiber bridges multiprocessing APIs with a flexible suite of backends that runs on various cluster management systems. It includes three layers (i.e. API layer, backend layer and cluster layer). The API layer provides the building blocks. The backend handles tasks (e.g. creating and terminating jobs). The cluster layer includes different cluster managers. For more information, check out the Fiber site.