Serverless Framework

Hi! My name is Stephen and as a GIP Intern I worked in the DevOps team at HDE. My primary work during my internship involved using AWS Lambda functions. AWS Lambda are based upon the idea of ‘Serverless’ architecture. This means you don’t need to have an active server listening to requests, and you don’t to worry about maintaining any server. Lambda functions also make it very easy to scale to a large amount of users without worrying about changing code. While Lambda functions provide amazing features, deploying and managing them is very difficult. This is especially true when there is a 3rd party library in your code. This requires a difficult zip packaging process, which has to be done every time you change your code.

To make this process simpler, I investigated tools to automate the deployment and management of Lambda functions. I tried many different 3rd party frameworks, but the most comprehensive and full featured framework was the Serverless Framework. With just a few commands, you can have your lambda function deployed to AWS. With Serverless Framework, the complicated zip packaging process is no longer needed. The Serverless Framework provides detailed documentation for setup, configuration, and 3rd party plugins. It allows easy management of AWS infrastructure, and has an extensive 3rd party community. For example, updating Lambda code is as simple as running:

$ sls deploy

Managing infrastructure with Serverless Framework is as simple as writing 1 or 2 lines. Included with every Serverless application is a yml file. YML files are a popular format for continuous integration services such as Travis CI and Circle CI, and its just as easy to use with Serverless. With the Serverless Framework, you can add and manage infrastructure associated with AWS Lambda functions such as SNS triggers, API Gateway, and S3 buckets. As an example, you can add an SNS trigger to your Lambda function by writing the following in your YML file:


After running the command sls deploy again, Serverless Framework will handle attaching this SNS trigger to your Lambda function automatically. This means you no longer have to navigate to the AWS console and manually add and create an SNS trigger. And then if you would like to remove this SNS trigger, it is as simple as removing that one line of code in the YML file. If you decide that you need to add API gateway to your Lambda function, Serverless makes that process very effortless. Just add the following code to your YML file: f:id:stephenlinkk:20171027153348p:plain

Then again, run: sls deploy. Serverless Framework will attach API Gateway to your Lambda function and display the URL associated with the API gateway. When you are done with the Lambda functions, it is very easy to remove your Lambda functions and the associated infrastructure, just run this command:

$ sls remove

After this command is run, you will see that the Lambda functions are now gone from AWS, and any SNS or API Gateway infrastructure is removed as well. I hope this brief introduction to the Serverless Framework encourages you start moving your application to Serverless architecture. If you would like a more in-depth look at Serverless Framework, please check out there website. Also, I wrote many instructions to deploy and manage Serverless applications. So please, check out my Gist!



Elasticsearch: the One Stop Shop for All Your Search Needs

Here at HDE, we started with a homemade indexing algorithm to create indices of data that would be searched by users of our service. But after several years of providing the service, we realized that the performance of this algorithm, and the system that came with it, was not built for the sheer scale of our growing userbase. We especially noticed this with some users whose data came in the scale of terabytes per day and demanded peak performance, which is what prompted us to finally look for an alternative in indexing and searching emails, and that's how we met Elasticsearch.


38th Monthly Technical Session (MTS) Report

38th Monthly Technical Session (MTS) was held on September 22nd, 2017. MTS is a knowledge sharing event, in which HDE members present some topics and have QA sessions, both in English.

The moderator of the 38th MTS was Kevin-san.


The first topic was "Introduction to AWS SAM" by Bagus. AWS SAM stands for AWS Serverless Application Model. As the name implies, it is a model used to define serverless applications on AWS. Serverless applications are applications composed of functions triggered by events.

AWS SAM is based on AWS CloudFormation, which is a service that allows users to manage AWS resources. AWS CloudFormation uses templates as blueprints for creating AWS resources. It manages related AWS resources as a single unit called a stack. With AWS SAM, a serverless application is defined in an AWS CloudFormation template and deployed as an AWS CloudFormation stack.

AWS SAM builds upon AWS CloudFormation to provide simpler ways to create AWS resources related to serverless applications. It provides new resource types, event source types, and property types. Being based on AWS CloudFormation should make AWS SAM work well with any serverless application frameworks that support AWS CloudFormation. Another advantage of using AWS SAM is AWS SAM Local, an AWS CLI tool for managing serverless applications written with AWS SAM. One of its main features is the ability to test AWS Lambda functions locally.


The second topic was "Elasticsearch: You Know, for Search" by Bumi-san. Elasticsearch is a distributed, RESTful search and analytics engine. It can be used for a broad variety of use cases, from simple keyword search to log aggregation and geolocation queries.

Elasticsearch is highly scalable, highly available, and is an all-in-one toolbox. It runs well both on a laptop and on a cluster of hundreds of servers handling petabytes of data. According to Bumi-san, in order to achieve desirable availability, it provides automatic recovery and data replication. It also comes with features such as aggregations, suggestions, and on the latest version: machine learning.

Elasticsearch provides full text search, and this feature is built on solid text analysis capabilities. Elasticsearch has no shortage of text analysis tools, such as analyzers, tokenizers, stemmers, and more. It also handles stopwords, synonyms, and misspellings.


The third topic was "PyCon APAC 2017" by Doi-san and Yuri-san. The event had two keynotes. The first keynote was about Python's impact on the business world. The second keynote was about Python community. It focused on several aspects, one of which is teaching people to program.

In total, there were 29 sessions from 27 speakers. A good number of these sessions are about topics that are trending recently, such as artificial intelligence, machine learning, data analytics, and big data. There were also several sessions about Python community. Based on these sessions, Doi-san concluded that Python 3 is still underutilized.

HDE was a proud sponsor of the event. We set up a booth and interacted with the attendees. Most of the nearly 200 attendees of the event are on jobs. Interestingly, almost half of the attendees came from overseas.


The fourth topic was "Introduction to Landscape" by Kusumoto-san. Landscape is a management tool to deploy, monitor, and manage Ubuntu servers. Landscape is available both on-premise and as a software-as-a-service. Landscape On-premises is free (for up to 10 machines), while Landscape SaaS is a paid service.

Landscape provides quite a lot of features, such as systems management, monitoring, security and compliance maintenances, inventory control, and package repository management. Kusumoto-san explained how to install Landscape client and demonstrated each of the aforementioned features.


The fifth topic was "Join the Dark Side (What Is Metaprogramming)" by Stefan-san. He is one of our Global Internship Program (GIP) participants. Metaprogramming is a programming technique in which computer programs have the ability to treat programs as their data. Consequently, a program can be designed to generate, read, or transform other programs. Metaprogramming can even allow a program to modify itself during runtime.

Metaprogramming is done in different ways across multiple programming languages. Stefan-san explained how metaprogramming is done in Python, Java, Go, and Ruby. He had actually used metaprogramming before, in Ruby. He mentioned some use cases of metaprogramming, one of which is 'redirecting' non-existent functions to existing ones.


The sixth topic was "Why I Love React" by Elvan-san. He is also one of our GIP participants. React is a JavaScript library for building user interfaces. At a glance, React is the view part of web applications, encourages declarative user interfaces, and drives component-based development. React has some unique aspects, such as components, JSX, and virtual DOM.

Elvan-san explained the aspects of React he likes the most. Moving between projects is easy due to common React concepts. There are apparently a lot of React components available. React also has hot reload. But perhaps more than all of these, he likes that React enables maintainability and scalability. In other words, making changes is straightforward and adding features is simple due to straightforward architecture.


The seventh topic was "Startup Incubators in USA: Chasing the American Dream!" by Stephen-san. He is also one of our GIP participants. Startup incubators are companies that help startups by providing services such as management training or office space. Through incubators, startups gains 'seed' funding, advice, mentorship, networking opportunities, strong community, and friendships, among other things.

Stephen-san told the story of his participation in a startup incubator program in his university. Non-students can participate as well, and in total there were 9 startups. The program was 10 weeks long, during which they have activities such as visiting venture capitals and investors. He also introduced his startup from the program, including the members and the product.


As usual, we had a party afterwards :)

37th Monthly Technical Session (MTS) Report

37th Monthly Technical Session (MTS) was held on August 25th, 2017. MTS is a knowledge sharing event, in which HDE members present some topics and have QA sessions, both in English.


The moderator of the 37th MTS was David-san.


The first topic was "AWS Service Update Summary 2017 Q2+ (April - August)" by Mitsuharu Hamba-san from AWS. He began by sharing information on new regions. A new AWS Region in Paris will be opened in 2017. Another new AWS Region in Stockholm will be opened in 2018. AWS will also open a new region in Osaka in 2018. According to Hamba-san, Osaka will be a local region. It is assumed to be used in combination with Tokyo region.

Some existing services had also been made available for Tokyo region in the last 4 months. First is Amazon EC2 P2 instances, which are ideal for compute-intensive applications that require high-performance GPU coprocessors and massive parallel floating point performance. Next is Amazon Lightsail, which helps the launch and management of virtual private servers. In the case of AWS X-Ray, it had moved from preview to general availability.

There are so many updates to existing services. Detailed, well-written information on them are readily available at the AWS Blog.


The second topic was an explanation of the data collection flow of our company's data warehouse project by Kogure-san. This project that he has been working on is very important in enabling data visualization and environment analysis. This project consists of several steps, but in this session he focused on explaining the first one, which is building data collection flow.

This data warehouse project collects HDE One services data. Each HDE One service has their own data collection flow. Kogure-san explained all of the completed ones. He showed the architectures and reported the number of records of each data collection flow. Now that Amazon Kinesis Firehose is available in the Tokyo region, he would like to utilize it to improve his current designs.


The third topic was "How to Find DMARC Failure" by Okubo-san. Okubo-san has been teaching us about DMARC for some time now, presenting topics about it on the 26th and 27th MTS. He also wrote an article about it on this blog. This time, he explained a project he's working on which monitors and visualizes DMARC reports.

Brief review: DMARC is an email authentication, policy, and reporting protocol. Its authentication is based on SPF and DKIM. ISPs provide DMARC reports for email senders. If a DMARC record includes rua parameter, then DMARC reports can be received via email. However, DMARC reports are XML files, so they are not exactly human-readable. Okubo-san's project summarizes and presents DMARC reports so that they are easier to understand. Some of its features include identifying DKIM and SPF results and showing WHOIS information.


The fourth topic was "Let's Warm up IP Addresses" by Matsuura-san. IP reputation is very important for message transfer agents (MTAs). Sending email, especially lots of them, from a brand new IP address is not advantageous. The spike of volume of emails sent from an IP address may result in it getting a bad reputation. Emails sent from an IP address with bad reputation will likely fail to reach their destinations.

One way to solve this is by warming up IP addresses, gradually increasing the volume of emails sent from IP addresses according to a predetermined schedule. Matsuura-san explained some problems of warming up IP addresses to make them work with Office 365. The first problem is receiving limits, which is the number of emails that can be received per hour. The limits are considerably less than his target, so he utilized shared mailboxes to receive more emails. The second problem is the opacity of Office 365's IP throttling methods. This is solved by warming IP addresses for several weeks, and stop increasing volume of emails sent for a few days whenever an IP address was throttled.


The fifth topic was "3 Interesting Facts about Filipinos" by Furukawa-san. She had been learning English in Cebu, Philippines, for almost 4 months. She told us some facts that are interesting to her. First, Filipinos seem to love karaoke so much. She saw people singing while walking on the streets and restaurant and shop staffs singing while working. There seem to be many karaoke shops and machines in the Philippines, as she showed some pictures of them.

Second, Filipinos seem to love to take selfies. When Furukawa-san went to tourist attractions with her Filipino teachers, they spent 1 to 2 hours taking selfies. TIME magazine awarded Makati City, Philippines, as the Selfie Capital of the World.

Third, Filipinos celebrate Christmas for 4 months. Christmas season starts from September to December, the so-called 'Ber Months'. Most Filipinos are Christians, and Filipinos seem to love celebrations. When September comes, Filipinos decorate their houses and shops, and Christmas songs can be heard in many places.


The sixth topic was "Auditing with Lighthouse" by Kevin-san. Lighthouse is an open-source, automated tool for improving the quality of web pages. It analyzes web apps and web pages, collecting modern performance metrics and insights on developer best practices. It has been a part of Chrome DevTools since Chrome 60. So perhaps the simplest way to check it out is to update to Chrome 60 and click on Chrome DevTools' Audits toolbar.

Lighthouse scores 4 categories: Progressive Web App, Performance, Accessibility, and Best Practices. Progressive Web App audit checks whether a site or app is interactive online. Performance audit refreshes a site or app with the new 'Slow 3G' network throttle. Accessibility audit checks for ARIA roles, uses aXe. Best practices audit checks for manifest.json files, checks for passive event listeners, etc.


The seventh topic was "EuroPython 2017" by Jonas-san. EuroPython 2017 was held from 9th until 16th July, 2017, in Rimini, Italy. It was organized by the European Python Society with the help of 25 sponsors, attended by more than 1000 people. In Jonas' opinion, when compared to PyCon US, EuroPython is a lot smaller, a lot less commercial, a lot less professional, but a lot more enjoyable.

Some of the talks that Jonas found interesting includes "A Python for Future Generations" by Armin Ronacher; "Making Games with Python: Mission Impossible?" by Tomislav Uzelac, Martin Christen, and Roberto De Ioris; "Fighting the Controls: Tragedy and Madness for Programmers and Pilots" by Daniele Procida; "PyPy Meets Python 3 and NumPy" by Armin Rigo; and "The Encounter: Python's Adventures in Africa" by Daniele Procida and Aisha Bello. Jonas himself had a talk, titled "Why You Might Want to Go Async". Besides talks, EuroPython also had social events, sprints (gathering in a room developing anything), and a hallway track.


As usual, we had a party afterwards :)


Attending de:code 2017 in Tokyo / de:code 2017 参加レポート

After a big conference Microsoft Build 2017 at Seattle, Washington in US in the beginning of May (my colleagues had written a report of Build), Microsoft also held another conference, de:code 2017, at The Prince Park Tower Tokyo in Japan for 2 days from May 23 to May 24, 2017. I would like to share my experience of attending it.

去る5月上旬にシアトルにて行われたMicrosoft Build 2017 (Buildについては同僚が参加レポートを書いてくれています)から約2週間後、ザ・プリンス パークタワー東京にてde:code 2017が開催されました。今回これに参加してきたので、体験して来たことを参加レポートとして共有したいと思います。


de:code is an annual conference held by Microsoft, which is aimed for developers same as Build. However, unlike Build, this conference is more focused on Japan’s market and Japanese developer communities. This focus is reflected in its contents. So I would like to focus on the difference between Build and de:code in this report. If you want to know more details about de:code, videos are now available.

de:codeはBuildと同じくマイクロソフト公式の開発者向けカンファレンスではあるのですが、より日本の市場や開発者コミュニティの志向を反映したようなコンテンツでした。最大8トラック並行で様々なセッションがあり、詳細はde:codeの公式ブログにて確認できます。また、Channel 9にて動画も公開されています。


36th Monthly Technical Session (MTS) Report

36th Monthly Technical Session (MTS) was held on July 21st, 2017. MTS is a knowledge sharing event, in which HDE members present some topics and have QA sessions, both in English.


The moderator of the 36th MTS was Iskandar-san.


The first topic was "Machine Learning: Intuition" by Nutt-san. He mainly focused on supervised learning. There are two phases of supervised learning, training and testing. Given input-output pairs, a good mapping from input to output is identified in the training phase. This mapping is used to predict new inputs in the testing phase. A predictor should have the smallest error possible on test data (not training data).

Nutt-san also emphasized that supervised learning works on the base of correlation, not causation. A predictor correlates input to output without knowing about causation, so we have to select input features carefully.

Nutt-san also explained the difference between deductive reasoning and inductive reasoning. To put it simply, in deductive reasoning, a conclusion is reached by applying general rules. On the other hand, in inductive reasoning, a conclusion is reached by extrapolating specific cases. Deductive reasoning is always correct, while inductive reasoning is not always correct. Machine learning is a kind of inductive reasoning. In relation to this, he reminded us that no algorithm works best for all supervised learning problem.


The second topic was "Spurious" by Fukutomi-san. Threads are utilized quite extensively in a project he was working on. Threads are usually executed concurrently and share resources. Sometimes, multiple threads accessing the same resources is not preferable due to concurrency issues.

In Java, one way to solve this is to synchronize threads. Another way is to utilize guarded blocks, which involves methods such as wait() and notify(). Unfortunately, there was a problem when Fukutomi-san was working with guarded blocks. It turned out that a thread can also wake up without being notified, interrupted, or timing out. This is called a spurious wakeup. He worked around this limitation by utilizing a true_wakeup flag.


The third topic was an explanation of a new component of an HDE service by Ogawa-san. He began by explaining the role of the new component in the HDE service. Then, he explained the technologies involved in the development of the new component. He developed the component using C++14 and Windows API, and he developed the installer program using C# 7 and .NET Framework 4.6.

Ogawa-san had to use C++ due to the component's relationship with Windows' Local Security Authority Subsystem Service (LSASS). High-level features can not be used in core operating system processes such as LSASS. In his opinion, reporting events to the Event Viewer from C++ code is not ideal. He also explained his approaches to unit test and continuous integration.


The fourth topic was "Security Assessment with Amazon Inspector" by Jeffrey-san. Amazon Inspector is an automated security assessment service that helps improve the security and compliance of applications deployed on AWS.

Jeffrey-san explained how to use the service. First, Amazon Inspector AWS Agents are installed in the target Amazon EC2 instances. Second, Amazon Inspector Assessment Targets, which are collections of EC2 instances to be scanned, are defined. Third, Amazon Inspector Assessment Templates, which defines the standardized tests to be applied on the assessment targets, are created. Finally, the assessment is run.

Amazon Inspector handles the analysis and even generates its reports. Alternatively, findings from Amazon Inspector can also be retrieved via APIs. This allows users to generate and format their own summarized or detailed reports.

Some pros to using Amazon Inspector are AWS nativity, low cost (30 cents per AWS Agent per assessment), and it is a good option for analysis of infrastructure vulnerabilities. Some cons to using Amazon Inspector are the limitation to EC2 instances and some benchmarks only work for certain operating systems.


The fifth topic was "But Will It Compile in Space!?" by Ignaty-san. He was one of our Global Internship Program (GIP) participants. This topic is a look at the effects of space radiation on electronics. There are several major radiation sources in space, such as solar wind, Van Allen Belts, changes with solar weather, and cosmic rays.

Radiation in space is much harsher than radiation on Earth. At such harsh levels, radiation can cause several kinds of damage to electronics. It can induce single-event effects which result in data degradation, calculation or logic errors, and any number of malfunctions. It can also cause gradual component degradation, which results in certain components failing entirely.

There are some ways to mitigate the effects of space radiation on electronics. The classic solution is radiation hardening. This essentially means components are made from more durable materials, which is expensive. Other solutions consists of avoiding radiation belts, shielding electronic components, designing fault-tolerant software and utilizing redundant components.


The sixth topic was "The Sweets and Bitters of React Native" by Rachel-san. She was also one of our GIP participants. React Native is a framework for building native apps using React. The motivation behind it is the desire to write mobile apps with the same logic as web apps, while achieving native behavior, without sacrificing performance. It reuses React logic in app development, is a bridge to native APIs, and executes JavaScript on the background thread.

Some pros to using React Native are easy to pick up for web developers, provides shared logic and code base for iOS and Android, gets rid of heavy IDEs, provides hot reloading, and easy to combine with native codes. Some cons to using React Native are knowledge of mobile native platform is required, relies on third-party libraries and documentations, frequent release cycles, and many ongoing problems due to its relative immaturity.


The seventh topic was "TensorFlow - Machine Learning without PhD" by Dovile-san. She was also one of our GIP participants. TensorFlow is an open-source software library for machine intelligence. TensorFlow offers lots of speed with less computing power, uses data flow graphs for numerical computations ,and provides API for Java, C++, Python, and Go. Other TensorFlow-related features include TensorBoard for visualization and TensorFlow Research Cloud for computational resource.

Dovile-san demonstrated the usage of TensorFlow to build artificial neural networks. Given the MNIST database of handwritten digits, the task is to train a model to look at images and predict what digits they are. Using TensorFlow, she defined the number and shape of the layers of the neural network. She also specified the learning rule and error measure calculation.


As usual, we had a party afterwards :)


builderscon tokyo 2017 参加レポート (セッション/コーヒーカップ裏話/当日スタッフ/懇親会) #builderscon

2017/08/03から2017/08/05までの3日間開催されたbuilderscon tokyo 2017に、弊社エンジニア複数名がスポンサー企業枠または当日スタッフとして参加しました。


  • 気になったセッションのレポート
    • The Evolution of PHP at Slack HQ
    • 真のコンポーネント粒度を求めて
    • Factory Class
  • コーヒーカップ裏話
  • 当日スタッフ
  • 懇親会