What is Namespace Package

If several packages share the same root folder, then the root folder is a namespace package. subpackageA and subpackageb can be installed separately, even in different Python path, but they can be imported as importing a single package: import root.

Recently, I want to move some files from my Mac to NAS to save space. I need a application behave like Dropbox, which can show all the files in the NAS and only download the files I need. I have tried the QSync, but it does not have thumbnails for cloud image and it does not have icons to show the file status. I also tried the Seafile, it’s a powerful application, which requires 4G RAM to run, and there is bug in the thumbnail. I used to have a Synology ARM NAS, the Synology Drive has all the features I need, so I want to run it on my QNAP NAS. After some research, I managed to run Synology and QNAP together on my NAS. Here is the guide.

pep517 defines two hooks: build_wheel and build_sdist, which is required to build the package from source. Each build backend must implement these two hooks. It makes it possible to create other build backend such as flit or poetry.

[build-system]
# Defined by PEP 518:
requires = ["flit"]
# Defined by this PEP:
build-backend = "local_backend"
backend-path = ["backend"]

Besides setuptools, there are some other build back-end such as hatchling and flit. You can find the example here: Python Packaging Uer Guide - Choosing a build backend

Python Command Line Arguments

By default, as initialized upon program startup, a potentially unsafe path is prepended to sys.path:

python -m: prepend the current working directory.

python script.py: prepend the script’s directory. If it’s a symbolic link, resolve symbolic links.

python -c and python (REPL): prepend an empty string, which means the current working directory.

You can remove these path with -P param.

The most important difference between these two functions is that import_module() returns the specified package or module (e.g. pkg.mod), while import() returns the top-level package or module (e.g. pkg). – https://docs.python.org/3/library/importlib.html#importlib.import_module

import itertools and from requests import exceptions can be translated to:

How to speed up Git Command

The performance of file system in WSL2 is poor, it takes a long time to run git status in a host’s repo. The solution is to use git.exe in Windows folder. You can add this into your bashrc:

Rockbox

The original firmware supports few audio format, it even can’t play FLAC. I install rockbox on it, which support FLAC and other format and I can transfer music without using iTunes or Finder. It also support theme and plugin, which makes it more powerful.

MacPod error

If you restore the iPod on macOS, it raises Warning: This is a MacPod, Rockbox only runs on WinPods. See http://www.rockbox.org/wiki/IpodConversionToFAT32 during installation. The easiest way to fix this is to restore it on Windows.

args.getargspec

getargspec is dropped in 3.11. You can easily replaced it with getfullargspec . It returns FullArgSpec(args, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations) instead of ArgSpec(args, varargs, keywords, defaults) So args, _, kw, _ = inspect.getargspec(fn) can be replaced by args, _, kw, *_ = inspect.getfullargspec(fn) However, getfullargspec is retained primarily for use in code that needs to maintain compatibility with the Python 2 inspect module API.

Basic configuration

The line ending on Windows is CRLF, on Linux is LF. To prevent the line ending issue, we can set core.autocrlf to true on Windows to let git convert CRLF to LF when commit, and convert LF to CRLF when checkout. It is automatically configured if you install git on Windows.

Download the File Dynamically

If the file can be download from URL or you can create a local HTTP server to share the file, you can download the file, use it and delete it in one RUN command. For example:

After some research, I figure out the cause. Some views does not close multiprocessing.Pool after using it. The problem disappears when I use Pool with with statement.

But I’m still interested in it and wrote some testing code. The script is run in Python 3.6.8 and produce similar result when using multiprocessing.ThreadPool.

This setting makes it possible to switch input method based on the context of cursor when entering insert mode.

sis

I’m using sis package with this configuration. You may need to install macism if you’re not using railwaycat/emacsmacport. More settings can be found in emacs-smart-input-source.

(sis-ism-lazyman-config
  "com.apple.keylayout.US"
  "com.apple.inputmethod.SCIM.ITABC")

(sis-global-cursor-color-mode t)
(sis-global-respect-mode t)
(sis-global-context-mode t)
(sis-global-inline-mode t)

fcitx

You can also install fcitx-remote for-osx and use cute-jumper/fcitx.el to do so. As homebrew no longer support some build options, you need to follow the install instructions in the GitHub repository to build fcitx.

So I bought this NAS last month, and I’m satisfied with it. Here are the advantages and disadvantages.

Advantages

1. High performance.

   It is equipped with J4125 quad-core 2.0 GHz processor, 8G RAM, two 2.5G Ports and 4 bays. Here is the spec. Although J4125 is not the fastest CPU in 2022(the newer model coming with N5105), it is still able to run several docker containers together, and I can even run Synology and Windows 10 inside build-in Virtualization Station.

   ]]>https://fromkk.com/posts/internet-account-keeps-coming-back-after-deletion-on-macos/Sat, 08 Jan 2022 00:09:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/internet-account-keeps-coming-back-after-deletion-on-macos/Today I tried to delete an inactive Internet account on system preference. It was deleted successfully but come back again after 20 seconds. This drives me nuts.

   I tried these methods, but none of them works.

   • Boot in safe mode, delete account.
   • Delete record in ZACCOUNT table in ~/Library/Accounts/Accounts4.sqlite.
   • Delete related items in Keychain Access app.

   Later, RedHatDude’s answer gives me a clue, it looks like a iCloud sync problem. I tried to delete the account on my 3 MacBooks together. Thank goodness! It does not show up again.

   ]]>https://fromkk.com/posts/how-to-disable-auto-strip-in-charfield-in-django/Sun, 19 Dec 2021 21:20:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/how-to-disable-auto-strip-in-charfield-in-django/In Django, when edit field in admin page or post data to forms, the leading and tailing whitespace in CharField and TextField are removed.

   The reason is strip=True parameter in forms.CharField, which is added in Djagno 1.9. You can see the discussion in django tiket #4960 and here is source code. models.CharField and models.TextField use formfield() to create form to interact with user, then both of them eventually create a forms.CharField

   ]]>https://fromkk.com/posts/using-jsonfield-before-django-3-dot-1/Sat, 11 Sep 2021 21:12:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/using-jsonfield-before-django-3-dot-1/In Django 3.1, Django support save python data into database as JSON encoded data and it is also possible to make query based on field value in JSONField. The detailed usage can be found here. If you are using older version and want to try this feature. Though there are many packages ported this function, I recommend django-jsonfield-backport.

   django-jsonfield-backport

   This package save data as JSON in database and also support JSON query. If your database meet the requirements (MySQL > 5.7, PG > 9.5, MariaDB > 10.2 or SQLite > 3.9 with JSON1 extension), you can use JSONField like Django’s native implementation.

   ]]>https://fromkk.com/posts/dynamic-allocate-executors-when-executing-jobs-in-spark/Sun, 18 Jul 2021 16:52:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/dynamic-allocate-executors-when-executing-jobs-in-spark/<p>I wrote a Spark program to process logs. The number of logs always changes as time goes by. To ensure logs can be processed instantly, the number of executors is calculated by the maximum of logs per minutes. As a consequence, the CPU usage is low in executors. In order to decrease resource waste, I tried to find a way to schedule executors during the execution of program.</p> <p>As shown below, the maximum number of logs per minutes can be a dozen times greater than the minimum number in one day.</p>https://fromkk.com/posts/improve-kafka-throughput/Fri, 28 May 2021 00:57:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/improve-kafka-throughput/Kafka is a high-performance and scalable messaging system. Sometimes when handling big data. The default configuration may limit the maximum performance. In this article, I’ll explain how messages are generate and saved in Kafka, and how to improve performance by changing configuration.

   Kafka Internals

   How does Producer Send Messages?

   In short, messages will assembled into batches (named RecordBatch) and send to broker.

   The producer manages some internal queues, and each queue contains RecordBatch that will send to one broker. When calling send method, the producer will look into the internal queue and try to append this message to RecordBatch which is smaller than batch.size (default value is 16KB) or create new RecordBatch.

   ]]>https://fromkk.com/posts/fix-error-cask-java-is-unavailable-in-homebrew/Sun, 07 Mar 2021 00:10:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/fix-error-cask-java-is-unavailable-in-homebrew/After update brew to latest version, when calling cask related command, it always outputs Error: Cask 'java' is unavailable: No Cask with this name exists., such as brew list --cask. However, the brew command works.

   After doing some research, I found Java has been moved to homebrew/core. This makes sense now. I installed java by cask, but it’s not available now and cask throw this error. If I uninstall java from cask, the error should disappear.

   ]]>https://fromkk.com/posts/timezone-in-jvm/Sun, 18 Oct 2020 23:49:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/timezone-in-jvm/I wrote a Scala code to get the current time. However, the output is different on the development server and docker.
   import java.util.Calendar
   
   println(Calendar.getInstance().getTime)

   On my development server, it outputs Sun Oct 18 18:01:01 CST 2020, but in docker, it print a UTC time.

   I guess it related to the timezone setting and do a research, here is the result.

   ]]>https://fromkk.com/posts/using-cibuildwheel-to-create-python-wheels/Wed, 29 Jul 2020 22:53:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/using-cibuildwheel-to-create-python-wheels/Have you ever tried to install MySQL-python? It contains the C code and need to compile the code while install the package. You have to follow the steps in this articles: Install MySQL and MySQLClient(Python) in MacOS. Things get worse if you are using Windows.

   Luckily, as new distribution format Wheel has been published in PEP 427.

   The wheel binary package format frees installers from having to know about the build system, saves time by amortizing compile time over many installations, and removes the need to install a build system in the target environment.

   ]]>https://fromkk.com/posts/retrieve-large-dataset-in-elasticsearch/Sun, 21 Jun 2020 20:33:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/retrieve-large-dataset-in-elasticsearch/It’s easy to get small dataset from Elasticsearch by using size and from. However, it’s impossible to retrieve large dataset in the same way.

   Deep Paging Problem

   As we know it, Elasticsearch data is organised into indexes, which is a logical namespace, and the real data is stored into physical shards. Each shard is an instance of Lucene. There are two kind of shards, primary shards and replica shards. Replica shards is the copy of primary shards in case nodes or shards fail. By distributing documents in an index across multiple shards, and distributing those shards across multiple nodes, Elasticsearch can ensure redundancy and scalability. By default, Elasticsearch create 5 primary shards and one replica shard for each primary shards.

   ]]>https://fromkk.com/posts/program-crash-caused-by-cpu-instruction/Sun, 17 May 2020 17:36:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/program-crash-caused-by-cpu-instruction/It’s inevitable to dealing with bugs in coding career. The main part of coding are implementing new features, fixing bugs and improving performance. For me, there are two kinds of bugs that is difficult to tackle: those are hard to reproduce, and those occur in code not wrote by you.

   Recently, I met a bug which has both features mentioned before. I write a Spark program to analyse the log and cluster them. Last week I update the code, use Facebook’s faiss library to accelerate the process of find similar vector. After I push the new code to spark, the program crashed. I found this log on Spark driver:

   ]]>https://fromkk.com/posts/c-m-ret-and-return-key-in-emacs/Sat, 11 Apr 2020 21:23:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/c-m-ret-and-return-key-in-emacs/I use Emacs to write blog. In the recent update, I found M-RET no longer behave as leader key in org mode, but behave as org-meta-return. And even more strange is that in other mode, it behave as leader key. And M-RET also works in terminal in org mode. In GUI, pressing C-M-m can trigger leader key.

   SO I opened this issue, with the help of these friends, the issue has been fixed. Here is the cause of the bug.

   ]]>https://fromkk.com/posts/import-custom-package-or-module-in-pyspark/Thu, 02 Apr 2020 22:24:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/import-custom-package-or-module-in-pyspark/First zip all of the dependencies into zip file like this. Then you can use one of the following methods to import it.

   |-- kk.zip
   |   |-- kk.py

   Using –py-files in spark-submit

   When submit spark job, add --py-files=kk.zip parameter. kk.zip will be distributed with the main scrip file, and kk.zip will be inserted at the beginning of PATH environment variable.

   ]]>https://fromkk.com/posts/time-boundary-in-influxdb-group-by-time-statement/Sun, 29 Mar 2020 22:30:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/time-boundary-in-influxdb-group-by-time-statement/These days I use InfluxDB to save some time series data. I love these features it provides:

   High Performance

   According to to it’s hardware guide, a single node will support more than 750k point write per second, 100 moderate queries per second and 10M series cardinality.

   Continuous Queries

   Simple aggregation can be done by InfluxDB’s continuous queries.

   Overwrite Duplicated Points

   If you submit a new point with same measurements, tag set and timestamp, the new data will overwrite the old one.

   ]]>https://fromkk.com/posts/c3-linearization-and-python-mro--method-resolution-order/Sat, 14 Mar 2020 17:37:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/c3-linearization-and-python-mro--method-resolution-order/Python supports multiple inheritance, its class can be derived from more than one base classes. If the specified attribute or methods was not found in current class, how to decide the search sequence from superclasses? In simple scenario, we know left-to right, bottom to up. But when the inheritance hierarchy become complicated, it’s not easy to answer by intuition.

   For instance, what’s search sequence of class M?

   class X:pass
   class Y: pass
   class Z:pass
   class A(X,Y):pass
   class B(Y,Z):pass
   class M(B,A,Z):pass

   The answer is: M, B, A, X, Y, Z, object

   ]]>https://fromkk.com/posts/difference-between-value-and-pointer-variable-in-defer-in-go/Thu, 19 Dec 2019 22:33:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/difference-between-value-and-pointer-variable-in-defer-in-go/defer is a useful function to do cleanup, as it will execute in LIFO order before the surrounding function returns. If you don’t know how it works, sometimes the execution result may confuse you.

   How it Works and Why Value or Pointer Receiver Matters

   I found an interesting code on Stack Overflow:

   type X struct {
       S string
   }
   
   func (x X) Close() {
       fmt.Println("Value-Closing", x.S)
   }
   
   func (x *X) CloseP() {
       fmt.Println("Pointer-Closing", x.S)
   }
   
   func main() {
       x := X{"Value-X First"}
       defer x.Close()
       x = X{"Value-X Second"}
       defer x.Close()
   
       x2 := X{"Value-X2 First"}
       defer x2.CloseP()
       x2 = X{"Value-X2 Second"}
       defer x2.CloseP()
   
       xp := &X{"Pointer-X First"}
       defer xp.Close()
       xp = &X{"Pointer-X Second"}
       defer xp.Close()
   
       xp2 := &X{"Pointer-X2 First"}
       defer xp2.CloseP()
       xp2 = &X{"Pointer-X2 Second"}
       defer xp2.CloseP()
   }

   The output is:

   ]]>https://fromkk.com/posts/near-duplicate-with-simhash/Wed, 04 Dec 2019 00:16:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/near-duplicate-with-simhash/Before talking about SimHash, let’s review some other methods which can also identify duplication.

   Longest Common Subsequence(LCS)

   This is the algorithm used by diff command. It is also edit distance with insertion and deletion as the only two edit operations.

   This works good for short strings. However, the algorithm’s time complexity is \(O(m*n)\), if two strings’ lengths are \(m\) and \(n\) respectively. So it’s not suitable for large corpus. Also, if two corpus consists of same paragraph but the order is not same. LCS treat them as different corpus, and that’s not we expected.

   ]]>https://fromkk.com/posts/jaeger-code-structure/Sun, 22 Sep 2019 17:07:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/jaeger-code-structure/Here is the main logic for jaeger agent and jaeger collector. (Based on jaeger 1.13.1)

   Jaeger Agent

   Collect UDP packet from 6831 port, convert it to model.Span, send to collector by gRPC

   Jaeger Collector

   Process gRPC or process packet from Zipkin(port 9411).

   Jaeger Query

   Listen gRPC and HTTP request from 16686.

   ]]>https://fromkk.com/posts/the-annotated-the-annotated-transformer/Sun, 01 Sep 2019 16:00:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/the-annotated-the-annotated-transformer/Thanks for the articles I list at the end of this post, I understand how transformers works. These posts are comprehensive, but there are some points that confused me.

   First, this is the graph that was referenced by almost all of the post related to Transformer.

   Transformer consists of these parts: Input, Encoder*N, Output Input, Decoder*N, Output. I’ll explain them step by step.

   Input

   The input word will map to 512 dimension vector. Then generate Positional Encoding(PE) and add it to the original embeddings.

   ]]>https://fromkk.com/posts/different-types-of-attention/Mon, 15 Jul 2019 00:16:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/different-types-of-attention/\(s_t\) and \(h_i\) are source hidden states and target hidden state, the shape is (n,1). \(c_t\) is the final context vector, and \(\alpha_{t,s}\) is alignment score.

   \[\begin{aligned} c_t&=\sum_{i=1}^n \alpha_{t,s}h_i \\ \alpha_{t,s}&= \frac{\exp(score(s_t,h_i))}{\sum_{i=1}^n \exp(score(s_t,h_i))} \end{aligned}\]

   Global(Soft) VS Local(Hard)

   Global Attention takes all source hidden states into account, and local attention only use part of the source hidden states.

   Content-based VS Location-based

   Content-based Attention uses both source hidden states and target hidden states, but location-based attention only use source hidden states.

   ]]>https://fromkk.com/posts/torchtext-snippets/Mon, 01 Jul 2019 21:28:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/torchtext-snippets/Load separate files

   data.Field parameters is here.

   When calling build_vocab, torchtext will add <unk> in vocabulary list. Set unk_token=None if you want to remove it. If sequential=True (default), it will add <pad> in vocab. <unk> and <pad> will add at the beginning of vocabulary list by default.

   LabelField is similar to Field, but it will set sequential=False, unk_token=None and is_target=Ture

   INPUT = data.Field(lower=True, batch_first=True)
   TAG = data.LabelField()
   
   train, val, test = data.TabularDataset.splits(path=base_dir.as_posix(), train='train_data.csv',
                                                   validation='val_data.csv', test='test_data.csv',
                                                   format='tsv',
                                                   fields=[(None, None), ('input', INPUT), ('tag', TAG)])

   Load single file

   all_data = data.TabularDataset(path=base_dir / 'gossip_train_data.csv',
                                  format='tsv',
                                  fields=[('text', TEXT), ('category', CATEGORY)])
   train, val, test = all_data.split([0.7, 0.2, 0.1])

   Create iterator

   train_iter, val_iter, test_iter = data.BucketIterator.splits(
       (train, val, test), batch_sizes=(32, 256, 256), shuffle=True,
       sort_key=lambda x: x.input)

   Load pretrained vector

   vectors = Vectors(name='cc.zh.300.vec', cache='./')
   
   INPUT.build_vocab(train, vectors=vectors)
   TAG.build_vocab(train, val, test)

   Check vocab sizes

   You can view vocab index by vocab.itos.

   ]]>https://fromkk.com/posts/build-your-own-tiny-tiny-rss-service/Mon, 10 Jun 2019 00:25:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/build-your-own-tiny-tiny-rss-service/After Inoreader change the free plan, which limit the max subscription to 150, I begin to find an alternative. Finally, I found Tiny Tiny RSS. It has a nice website and has the fever API Plugin which was supported by most of the RSS reader app, so you can read RSS on all of you devices.

   This post will tell you how to deploy it on your server.

   Prerequisite

   You need to install Docker and Docker Compose before using docker-compose.yml

   ]]>https://fromkk.com/posts/preview-latex-in-org-mode-with-emacs-in-macos/Sun, 12 May 2019 20:26:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/preview-latex-in-org-mode-with-emacs-in-macos/Using the right Emacs Version

   I failed to preview LaTeX with emacs-plus. If you have installed d12frosted/emacs-plus, uninstall it and use emacs-mac.

   brew tap railwaycat/emacsmacport
   brew install emacs-mac

   If you like the fancy spacemacs icon, install it with cask: brew cask install emacs-mac-spacemacs-icon

   Install Tex

   • Download and install BasicTeX.pkg here.
   • Add /Library/TeX/texbin to PATH.
   • Install dvisvgm by sudo tlmgr update --self && sudo tlmgr install dvisvgm collection-fontsrecommended

   Emacs settings

   • Add TeX related bin to path: (setenv "PATH" (concat (getenv "PATH") ":/Library/TeX/texbin"))
   • Tell Org Mode to create svg images: (setq org-latex-create-formula-image-program 'dvisvgm)

   Now you can see the rendered LaTeX equation by calling org-preview-latex-fragment or using shortcut ,Tx.

   ]]>https://fromkk.com/posts/using-ddqn-to-play-flappy-bird/Sun, 14 Apr 2019 17:10:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/using-ddqn-to-play-flappy-bird/PyTorch provide a simple DQN implementation to solve the cartpole game. However, the code is incorrect, it diverges after training (It has been discussed here).

   The official code’s training data is below, it’s high score is about 50 and finally diverges.

   There are many reason that lead to divergence.

   First it use the difference of two frame as input in the tutorial, not only it loss the cart’s absolute information(This information is useful, as game will terminate if cart moves too far from centre), but also confused the agent when difference is the same but the state is varied.

   ]]>https://fromkk.com/posts/circular-import-in-python/Sun, 10 Mar 2019 10:59:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/circular-import-in-python/Recently, I found a really good example code for Python circular import, and I’d like to record it here.

   Here is the code:

   # X.py
   def X1():
       return "x1"
   
   from Y import Y2
   
   def X2():
       return "x2"

   # Y.py
   def Y1():
       return "y1"
   
   from X import X1
   
   def Y2():
       return "y2"

   Guess what will happen if you run python X.py and python Y.py?

   ]]>https://fromkk.com/posts/python-dictionary-implementation/Sun, 17 Feb 2019 21:48:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/python-dictionary-implementation/Overview

   1. CPython allocation memory to save dictionary, the initial table size is 8, entries are saved as <hash,key,value> in each slot(The slot content changed after Python 3.6).
   2. When a new key is added, python use i = hash(key) & mask where mask=table_size-1 to calculate which slot it should be placed. If the slot is occupied, CPython using a probing algorithm to find the empty slot to store new item.
   3. When 2/3 of the table is full, the table will be resized.
   4. When getting item from dictionary, both hash and key must be equal.

   Resizing

   When elements size is below 50000, the table size will increase by a factor of 4 based on used slots. Otherwise, it will increase by a factor of 2. The dictionary size is always \(2^{n}\).

   ]]>https://fromkk.com/posts/textcnn-with-pytorch-and-torchtext-on-colab/Mon, 03 Dec 2018 15:47:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/textcnn-with-pytorch-and-torchtext-on-colab/PyTorch is a really powerful framework to build the machine learning models. Although some features is missing when compared with TensorFlow (For example, the early stop function, History to draw plot), its code style is more intuitive.

   Torchtext is a NLP package which is also made by pytorch team. It provide a way to read text, processing and iterate the texts.

   Google Colab is a Jupyter notebook environment host by Google, you can use free GPU and TPU to run your modal.

   ]]>https://fromkk.com/posts/csrf-in-django/Wed, 07 Nov 2018 13:58:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/csrf-in-django/CSRF(Cross-site request forgery) is a way to generate fake user request to target website. For example, on a malicious website A, there is a button, click it will send request to www.B.com/logout. When the user click this button, he will logout from website B unconsciously. Logout is not a big problem, but malicious website can generate more dangerous request like money transfer.

   Django CSRF protection

   Each web framework has different approach to do CSRF protection. In Django, the validation process is below:

   ]]>https://fromkk.com/posts/create-node-benchmark-in-py2neo/Mon, 05 Nov 2018 15:55:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/create-node-benchmark-in-py2neo/Recently, I’m working on a neo4j project. I use Py2neo to interact with graph db. Although Py2neo is a very Pythonic and easy to use, its performance is really poor. Sometimes I have to manually write cypher statement by myself if I can’t bear with the slow execution. Here is a small script which I use to compare the performance of 4 different ways to insert nodes.

   import time
   
   from graph_db import graph
   
   from py2neo.data import Node, Subgraph
   
   
   def delete_label(label):
       graph.run('MATCH (n:{}) DETACH DELETE n'.format(label))
   
   
   def delete_all():
       print('delete all')
       graph.run('match (n) detach delete n')
   
   
   def count_label(label):
       return len(graph.nodes.match(label))
   
   
   def bench_create1():
       print('Using py2neo one by one')
       delete_label('test')
       start = time.time()
       tx = graph.begin()
       for i in range(100000):
           n = Node('test', id=i)
           tx.create(n)
       tx.commit()
       print(time.time() - start)
       print(count_label('test'))
       delete_label('test')
   
   
   def bench_create2():
       print('Using cypher one by one')
       delete_label('test')
       start = time.time()
       tx = graph.begin()
       for i in range(100000):
           tx.run('create (n:test {id: $id})', id=i)
           if i and i % 1000 == 0:
               tx.commit()
               tx = graph.begin()
       tx.commit()
       print(time.time() - start)
       print(count_label('test'))
       delete_label('test')
   
   
   def bench_create3():
       print('Using Subgraph')
       delete_label('test')
       start = time.time()
       tx = graph.begin()
       nodes = []
       for i in range(100000):
           nodes.append(Node('test', id=i))
       s = Subgraph(nodes=nodes)
       tx.create(s)
       tx.commit()
       print(time.time() - start)
       print(count_label('test'))
       delete_label('test')
   
   
   
   def bench_create4():
       print('Using unwind')
       delete_label('test')
       start = time.time()
       tx = graph.begin()
       ids = list(range(100000))
       tx.run('unwind $ids as id create (n:test {id: id})', ids=ids)
       tx.commit()
       print(time.time() - start)
       print(count_label('test'))
       delete_label('test')
   
   
   def bench_create():
       create_tests = [bench_create1, bench_create2, bench_create3, bench_create4]
   
       print('testing create')
       for i in create_tests:
           i()
   
   
   if __name__ == '__main__':
       bench_create()

   Apparently, using cypher with unwind keyword is the fastest way to batch insert nodes.

   ]]>https://fromkk.com/posts/deploy-nikola-org-mode-on-travis/Sat, 03 Nov 2018 21:20:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/deploy-nikola-org-mode-on-travis/Recently, I enjoy using Spacemacs, so I decided to switch to org file from Markdown for writing blog. After several attempts, I managed to let Travis convert org file to HTML. Here are the steps.

   Install Org Mode plugin

   First you need to install Org Mode plugin on your computer following the official guide: Nikola orgmode plugin.

   Edit conf.el

   Org Mode will convert to HTML to display on Nikola. Org Mode plugin will call Emacs to do this job. When I run nikola build, it shows this message: Please install htmlize from https://github.com/hniksic/emacs-htmlize. I’m using Spacemacs, the htmlize package is already downloaded if the org layer is enabled. I just need to add htmlize folder to load-path. So here is the code:

   ]]>https://fromkk.com/posts/using-chinese-characters-in-matplotlib/Thu, 04 Oct 2018 15:53:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/using-chinese-characters-in-matplotlib/After searching from Google, here is easiest solution. This should also works on other languages:

   import matplotlib.pyplot as plt
   %matplotlib inline
   %config InlineBackend.figure_format = 'retina'
   
   import matplotlib.font_manager as fm
   f = "/System/Library/Fonts/PingFang.ttc"
   prop = fm.FontProperties(fname=f)
   
   plt.title("你好",fontproperties=prop)
   plt.show()

   Output:

   ]]>https://fromkk.com/posts/lstm-and-gru/Sun, 22 Apr 2018 14:39:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/lstm-and-gru/LSTM

   The avoid the problem of vanishing gradient and exploding gradient in vanilla RNN, LSTM was published, which can remember information for longer periods of time.

   Here is the structure of LSTM:

   The calculate procedure are:

   \[\begin{aligned} f_t&=\sigma(W_f\cdot[h_{t-1},x_t]+b_f)\\ i_t&=\sigma(W_i\cdot[h_{t-1},x_t]+b_i)\\ o_t&=\sigma(W_o\cdot[h_{t-1},x_t]+b_o)\\ \tilde{C_t}&=tanh(W_C\cdot[h_{t-1},x_t]+b_C)\\ C_t&=f_t\ast C_{t-1}+i_t\ast \tilde{C_t}\\ h_t&=o_t \ast tanh(C_t) \end{aligned}\]

   \(f_t\),\(i_t\),\(o_t\) are forget gate, input gate and output gate respectively. \(\tilde{C_t}\) is the new memory content. \(C_t\) is cell state. \(h_t\) is the output.

   ]]>https://fromkk.com/posts/models-and-architechtures-in-word2vec/Fri, 05 Jan 2018 15:14:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/models-and-architechtures-in-word2vec/Generally, word2vec is a language model to predict the words probability based on the context. When build the model, it create word embedding for each word, and word embedding is widely used in many NLP tasks.

   Models

   CBOW (Continuous Bag of Words)

   Use the context to predict the probability of current word. (In the picture, the word is encoded with one-hot encoding, \(W_{V*N}\) is word embedding, and \(W_{V*N}^{’}\), the output weight matrix in hidden layer, is same as \(\hat{\upsilon}\) in following equations)

   ]]>https://fromkk.com/posts/semi-supervised-text-classification-using-doc2vec-and-label-spreading/Sun, 10 Sep 2017 15:29:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/semi-supervised-text-classification-using-doc2vec-and-label-spreading/Here is a simple way to classify text without much human effort and get a impressive performance.

   It can be divided into two steps:

   1. Get train data by using keyword classification
   2. Generate a more accurate classification model by using doc2vec and label spreading

   Keyword-based Classification

   Keyword based classification is a simple but effective method. Extracting the target keyword is a monotonous work. I use this method to automatic extract keyword candidate.

   ]]>https://fromkk.com/posts/parameters-in-dov2vec/Thu, 03 Aug 2017 15:20:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/parameters-in-dov2vec/Here are some parameter in gensim’s doc2vec class.

   window

   window is the maximum distance between the predicted word and context words used for prediction within a document. It will look behind and ahead.

   In skip-gram model, if the window size is 2, the training samples will be this:(the blue word is the input word)

   min_count

   If the word appears less than this value, it will be skipped

   sample

   High frequency word like the is useless for training. sample is a threshold for deleting these higher-frequency words. The probability of keeping the word \(w_i\) is:

   ]]>https://fromkk.com/posts/brief-introduction-of-label-propagation-algorithm/Sun, 16 Jul 2017 21:45:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/brief-introduction-of-label-propagation-algorithm/As I said before, I’m working on a text classification project. I use doc2vec to convert text into vectors, then I use LPA to classify the vectors.

   LPA is a simple, effective semi-supervised algorithm. It can use the density of unlabeled data to find a hyperplane to split the data.

   Here are the main stop of the algorithm:

   1. Let $ (x_1,y1)…(x_l,y_l)$ be labeled data, $Y_L = \{y_1…y_l\} $ are the class labels. Let \((x_{l+1},y_{l+u})\) be unlabeled data where \(Y_U = \{y_{l+1}…y_{l+u}\}\) are unobserved, usually \(l \ll u\). Let \(X=\{x_1…x_{l+u}\}\) where \(x_i\in R^D\). The problem is to estimate \(Y_U\) for \(X\) and \(Y_L\).
   2. Calculate the similarity of the data points. The most simple metric is Euclidean distance. Use a parameter \(\sigma\) to control the weights.

   \[w_{ij}= exp(-\frac{d^2_{ij}}{\sigma^2})=exp(-\frac{\sum^D_{d=1}{(x^d_i-x^d_j})^2}{\sigma^2})\]

   ]]>https://fromkk.com/posts/enable-c-extension-for-gensim-on-windows/Sat, 10 Jun 2017 14:43:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/enable-c-extension-for-gensim-on-windows/These days, I’m working on some text classification works, and I use gensim ’s doc2vec function.

   When using gensim, it shows this warning message:

   C extension not loaded for Word2Vec, training will be slow.

   I search this on Internet and found that gensim has rewrite some part of the code using cython rather than numpy to get better performance. A compiler is required to enable this feature.

   I tried to install mingw and add it into the path, but it’s not working.

   ]]>https://fromkk.com/posts/some-useful-shell-tools/Sun, 07 May 2017 15:34:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/some-useful-shell-tools/Here are some shell tools I use, which can boost your productivity. Mordern-unix is a great repo that list lots of modern unix tools.

   Prezto

   A zsh configuration framework. Provides auto completion, prompt theme and lots of modules to work with other useful tools. I extremely love the agnoster theme.

   Fasd

   Help you to navigate between folders and launch application.

   Here are the official usage example:

   ]]>https://fromkk.com/posts/start/Tue, 18 Apr 2017 15:46:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/start/<p>Over the years, I have read so many programmers’ blogs, which has helped me a lot. Now I think it’s the time to start my own blog.</p> <p>I hope this can enforce myself to review what I have learned, and it would even be better if someone can benefit from it.</p>