Recently, I worked on a task which need to collect all CloudWatch logs to a Kinesis stream. The project is using Serverless for deployment. There are some plugins to create CloudWatch Log subscription filter, but none of them using Kinesis as the destination.
Then by using the serverless-scriptable-plugin, I’m able to do this very easily. The following code find out all CloudWatch LogGroups, and create a SubscriptionFilter for each of them.
I’m working on a project which needs to fetch messages from hundreds of SQS queues. We’re using SQS long polling to reduce the number of empty responses. It was very quick to get response at first when there are only dozen queues. As we added more and more queues, the performance getting worse and worse. It takes 60 seconds to get the response when there’s 300 queues and WaitTimeSeconds set to 10 seconds.
We are using Node.js in single thread mode, and I believe that it could handle 10 thousands connections without any problem because most of the tasks are IO processing. We also created an AWS support case, but nobody clearly answered the question.