Archived | Futa Waseda

Twitter Image Captioning

Tue, 05 Jan 2021 00:00:00 +0000

This work was done in the class activity at the university when I was a first-year master’s student.

We aimed to generate tweet-like text from an input image.

The difference between the general image captioning task is large. In the image captioning task, the purpose of the deep learning model is to explain the input image precisely and objectively. On the other hand, tweets are allowed to be subjective, vague, or humorous. Tweets can be diverse.

We first constructed Twitter Dataset using Twitter API. We collected about 5000 pairs of images and texts from 31 Twitter accounts. In this work, we only collected tweets of animals.

For model, we use Encoder(CNN)-Decoder(LSTM) model. In the training phase, we first pre-trained model by STAIR captions dataset for 10 epochs and then finetuned by Twitter dataset for 2 epochs.

For evaluation, we took a questionnaire from 35 participants. In the questionnaire, the naturalness of texts, the correspondence of image and text, humor, whether you want to LIKE the tweet were asked. The results showed that our model performed better than a model trained only by the STAIR dataset, yet it still loses to human’s real tweet.

Keywords.

Twitter API
python
pytorch
Image Captioning

Oshaberi-Bot（おしゃべりぼっと）

Wed, 01 Apr 2020 00:00:00 +0000

Oshaberi-Bot(おしゃべりぼっと:Talking bot) is my first twitter bot app. He learns japanese from his followers, by fitting retrieved data to Markov model.

Keywords.

Twitter API
python
Markov model

trip map

Mon, 30 Apr 2018 00:00:00 +0000

Trip map is the demo web application I made in school.

The purpose of this app is to make our trip planning more easy. In this app, you can clip the place you want to go in the future, find the shortest way to go through the chosen spots.

I was responsible for the front-end system. Front-end is written by html, css, javascript with jquery library. It was deployed by free heroku service.

Keywords.

html
css
javascript
jquery