Ideas for creating a friendfeed like feed aggregator system

Ideas for creating a friendfeed like feed aggregator system


Ideas for creating a friendfeed like feed aggregator system

I was invited to join a meeting about implement a friendfeed like feed system. Here are some ideas about requirement and architecture, which I typed on my BlackBerry during the meeting.

  1. Like the friendfeed, The product can import external RSS, so we separate the system into two parts. The rss crawl system and the feed pubsub system. The pubsub system has no responsbility to grab the feed from source. The feed itself only save feed summary and url.
  2. We decided to use the INBOX approach, which will push the published feed to be saved in all subscriber’s data table. More information of how this work can refer to Scaling a Microblogging Service – Part I.
  3. User’s homepage is an aggregation result. We choose to return a limited recent real-time date, no infinity pagination. But the user’s own feed(user’s profile page) may have a bigger date range.
  4. The unsubscibe logic have two choice, delete or keep the history data from one’s inbox. We decide to keep them.
  5. If the feed source had been deleted, do we need to delete all references in all subscriber’s inbox?  If need delete, each feed push to the pubsub system need to have a unique resource id. Another problem is after the source updated whether to publish a new feed or update the current feed?
  6. How to manage the group(QUN in Chinese) feed, deliver to all member’s inbox? Or share a group inbox?
  7. How to impl the feed comment logic, publish the comment to feed system or design a standalone comment system. We prefer to use a standalone comment system which doesn’t publish the comment back to the feed system.
  8. Every feed has a media type, such as text, video, image so the subscribe API can only retrieve a limited media type (text for mobile device). And a feed may have tags.
  9. The read/unread count is easy to implement. But the load is heavy. (QQ / QQzone may has such logic.)
  10. Need open API for 3rd party client(like twitter client), and RSS feed, may have OAuth integration.
  11. The storage may like friendfeed’s mysql schema (see How FriendFeed uses MySQL to store schema-less data) or use Amazon simpledb.
  12. May add support for XMPP Publish-Subscribe, or PEP(Personal Eventing Protocol) for pushing realtime time to users in the future.
好烂啊有点差凑合看看还不错很精彩 (No Ratings Yet)

4 thoughts on “Ideas for creating a friendfeed like feed aggregator system

  1. What i do not understood is in reality how you are now not actually a lot more well-appreciated than you may be now. You’re so intelligent. You understand thus significantly in the case of this topic, made me for my part consider it from so many numerous angles. Its like women and men are not interested until it?¦s something to do with Lady gaga! Your individual stuffs excellent. All the time deal with it up!

  2. Hi! I could have sworn I’ve been to this site before but after browsing through some of the post I realized it’s new to me. Anyways, I’m definitely delighted I found it and I’ll be bookmarking and checking back often!


电子邮件地址不会被公开。 必填项已用*标注