Twitter API Best Practices: DB Lists vs. Twitter Lists

Guest Author
Jul. 17 2012, 07:00AM EDT

This guest post comes from Adam Green, a Twitter API consultant who builds custom Twitter apps for clients, and blogs about Twitter programming at 140dev.com. Follow him at @140dev.

When Twitter API developers need to work with a list of Twitter accounts, the first instinct is to store them in a database table. I've been doing a lot of work with Twitter lists over the last few months, and I've come to realize that in some cases they have some real advantages over database lists. This column will compare the two models, and also propose a hybrid approach that combines the best of both. Let's start with some benefits of Twitter lists that may not be obvious to everyone.

Get aggregated tweets for an entire list

This was a real surprise for me when I found it, and a huge benefit of lists. You can make a single API call with /lists/statuses, and Twitter will return the most recent tweets for all members of a list. This is way faster and easier than having to retrieve the timelines of each member separately, store them in a database, and then pull them together in a single stream. If you have a couple of hundred accounts you need to display as a single tweet stream, this is a neat way of solving that problem.

Free UI for managing list of accounts

Being a database programmer, I always find UI coding a time consuming task. Twitter lists already have a set of free functionality built into Twitter.com for adding to and deleting from a list. You also have a built-in UI for searching for the account of someone you want on your list. Letting users create their own set of accounts in a Twitter list is so much easier than having to build this yourself with a database, and matching PHP/HTML/CSS/Ajax code.

Twitter lists can be shared between users

Users can have needs for the list of accounts beyond the features delivered by your code. If someone asked me to build an app based on a list of users, and also asked for an easy way to let users pass that list around, it would be a lot of extra work if the info was in a database. If the accounts are in a Twitter list, however, there is already a subscription mechanism available.

Database list wins on speed and storage

It should be obvious that delivering a list of Twitter accounts to your app will be faster if they are already in your database. I would not write an app that had to call the Twitter API every time it needed this info. Storage is another obvious problem. You can only have 500 members in a Twitter list. That isn't such a serious problem when you consider that the limit of 20 lists gives you a total of 10,000 accounts. If there is a logical model for segmenting users into separate lists, the storage problem may not affect you. Of course, spreading the accounts into too many lists makes it more difficult for users to edit them by hand.

API rate limits are a killer if repeated access is needed

It always comes down to rate limits, doesn't it? Even if you could get the contents of a Twitter list fast enough to use them in your code, API rate limits will be a serious problem. I'd hate to use up hundreds of API calls an hour getting the current copy of a list every time I needed it.

Hybrid solution: Twitter list and database list synched periodically

I think the best solution is to combine Twitter lists and database lists to give you the benefits of both. You can allow users to edit Twitter lists by hand, and then import them into a database on a regular basis with a cron job. You can also give the user an import option they can run from an admin page in your app after they edit a list on Twitter.com. This synchronized model lets you make the /lists/statuses API call when needed based on the Twitter lists, and use the Twitter list as a user input mechanism and a way for them to share the list. Other than getting the aggregated tweet stream, your code could get the data from the database list when responding to any other queries.

Use caution when synching back to Twitter

Reading from a Twitter list into a database is straight-forward, and can be done with the /lists/members API call. Where you have to be careful is writing from a database back to a Twitter list. If you plan on doing this, there are a couple of things to watch out for. The fast way to update a Twitter list seems to be to delete the list with a single API call using /lists/destroy to get rid of the current members, recreate a new copy of the list with /lists/create, and then add a new set of members using /lists/members/create_all. This is fast, and uses a minimum number of API calls. The weakness of this approach is that when you delete a list, all the subscribers get disconnected. If sharing the list among multiple users is important, that is a deal breaker.

The alternative method that will retain a list's subscribers is to keep the list, and delete all the members with /lists/members/destroy_all. Then you can add the new set of members back in. In practice I find that Twitter gets a little confused by this, and won't display the new list members for a while. If Twitter is overloaded, it can be a long while. This probably has something to do with Twitter's distributed database model. There are multiple copies of the list in different caching servers, which leads to confusion.

So if you do need to synch from a database to a Twitter list, be sure to test this part of the system carefully. That is the only limitation I know of in the hybrid model I've proposed here.

Guest Author

Comments

User HTML

  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.
  • Web page addresses and e-mail addresses turn into links automatically.