Stack Your Logs In One Place With Loggly

If you’re writing a serious application, on the web or otherwise, you’ve got to have good logging.  In the context of application development, reading logs is my personal favorite method for tracking bugs and rooting them out.  A service like Loggly is perfect for programmers like me.   It provides an easy to use web interface with searching capabilities that allows you to see what your application is logging at any time that you’d like to check in.  Debugging with Loggly is nice, but it’s just the tip of the iceberg, thanks in part to its Loggly API.

Imagine that you’re a web programmer implementing your own "passion project."  You start with one server and one application instance, but as the internet starts to show some traffic-love to your idea expansion becomes a pressing need.  There are plenty of cloud services out there that can replicate your application as needed, but that replication doesn’t solve the problems of coordination or analysis.  Once you have multiple instances of your application, how do you track what’s going on in any efficient way?  Loggly is your answer.  Consolidate your logs in one place and write your analytics to crunch that one data store.

I recently spoke with Kord Campbell, Loggly's CEO and founder about his vision for the company.  When I asked him what he saw as Loggly's ideal use case scenario he told me that he could see Loggly in literally hundreds of thousands of different applications.  Loggly's aim will be to keep the service simple enough that it can be relevant to as many use cases as possible.  Certainly that's the thing to do when you're building a utility for the web.

Loggly is currently humming along with average daily load from a typical account hitting 8GB worth of log messages, with the big ones coming in at 20GB, and some outliers pushing the envelope with 100GB days here and there.  That's a lot of data and it adds up quickly, which is part of Loggly's value proposition: let Loggly take care of log retention so you don't need to.  The next step beyond that is analysis.  Loggly presents a nice UI for searching and analyzing your logs over a period of time and there are more data crunching features in the works.

The Loggly RESTful API serves up JSON responses and offers two modes:   administration and dealing with logs.   To use either, you’ll have to gain clearance through OAuth.  This makes it a little harder to test the API from the command-line, but controlled access really is what you want, isn’t it?

In administration mode you can control both devices and inputs.  What are devices and inputs?  Two essential concepts that you need in order to understand loggly!  Devices send information to inputs.  This is a many to many relationship.  Under this model, you can have multiple devices sending data to a single input.  You can also have devices that send logs to multiple inputs.  Reflecting on this concept, one potential configuration could be a web application using one loggly input and one device to start.  If the application needs to be replicated due to high traffic, the new application instance could use the API to set up a new device on the same input.

With the API you can add, update, or delete inputs. You can even manage an input’s devices.  When it comes to devices, you can retrieve a list of all devices, and add or remove a device from an input.  Support for this type of administration through the API is a key to enabling new application instances to contact loggly to initiate new devices on existing inputs.

If you’re ready to dive in, check out the API docs, the NodeJS implementation of the API in Winston, or the node-loggly package, as well as the loggly team’s page on github.  Loggly is enjoying lots of interest from the development community as shown by the ever increasing amount of open source API implementations.

Be sure to read the next API article: Shared Count: A 6 Sided Popularity Metric API