redis-lua

Redis 設(shè)計與實現(xiàn)

Lua 腳本功能是 Reids 2.6 版本的最大亮點， 通過內(nèi)嵌對 Lua 環(huán)境的支持， Redis 解
決了長久以來不能高效地處理 CAS （check-and-set）命令的缺點， 并且可以通過組合使
用多個命令， 輕松實現(xiàn)以前很難實現(xiàn)或者不能高效實現(xiàn)的模式。

上文基本就是對Redis Lua scripting 的蹩腳翻譯。

本篇先介紹一下服務(wù)端腳本是如何使用的，然后對 redis 用于腳本支持的代碼進(jìn)行分析，
以了解如何在服務(wù)器中嵌入 lua 虛擬機(jī)和如何和其進(jìn)行交互。

Usage

運行腳本的命令：

• EVAL – used to evaluate scripts using the Lua intepreter built into Redis.

  • The first argument is a Lua 5.1 script. The script does not need to define
    a Lua function (and should not). A chunk, in Lua terminology.

    eval “l(fā)ocal function f() return 1 end return f()” 0
    (integer) 1

  • The second argument is the number of arguments that follows the script.
    The arguments can be accessed by Lua using the global variable KEYS.

  • The rest of the arguments can be accessed by Lua using the global variable
    ARGV.

  • Using redis.call() and redis.pcall to call Redis commands from a Lua
    script. The arguments of the redis.call() and redis.pcall() functions
    are simply all the arguments of a well formed Redis command.

    eval “return redis.call(‘set’, KEYS[1], ARGV[1])” 1 foo bar
    OK

  • Lua scripts can return a value, that is converted from the Lua type to
    the Redis protocol using a set of conversion rules. If a Redis type is
    converted into a Lua type, and then the result is converted back into a
    Redis type, the result is the same as of the initial value.

    • There is not simple way to have nils inside Lua arrays, this is a
      result of Lua table semantics, so when Redis converts a Lua array into
      Redis protocol the conversion is stopped if a nil is encourtered.

• EVALSHA – used to evaluate scripts using the Lua interpreter built into
  Redis. EVALSHA works exactly like EVAL, but instead of having a script as
  the first argument it has the SHA` digest of a script.

  • If the server still remembers a script with a matching SHA1 digest, the
    script is executed.

  • If the server does not remember a script with this SHA1 digest, a special
    error is returned telling the client to use EVAL instead.

Redis 使用唯一 Lua 解釋器運行所有腳本，并且保證腳本執(zhí)行的原子性：腳本正在運行期
間，Redis 不會執(zhí)行任何其它命令。

另外一方面，對腳本原子性的保證，在執(zhí)行較慢的腳本時，會降低整體的吞吐率。

管理腳本的命令：

• SCRIPT FLUSH – only way to force Redis to flush the script cache.
• SCRIPT EXISTS sha1 – check whether the scripts are still in Redis’s cache.
• SCRIPT LOAD script – register the specified script without executing it.
• SCRIPT KILL – only way to interrupt a long-running script that reach the
  configured maximum execution time for scripts.

如果腳本運行時間超出設(shè)置的最大運行時長后，Redis 開始接收并處理 SCRIPT KILL 和
SHUTDOWN NOSAVE 命令。

SCRIPT KILL 只能用來停止只執(zhí)行了 讀操作 的腳本。如果腳本已經(jīng)執(zhí)行了寫操作，客
戶端只能使用 SHUTDOWN NOSAVE 命令來關(guān)閉 Redis 服務(wù)端進(jìn)程以保證磁盤數(shù)據(jù)的一致
性。

Limitation

Redis 中的腳本會被復(fù)制到其它 slave 上，同時也會被原樣寫入 AOF 文件。這樣做的好處
是節(jié)省了帶寬 (直接傳輸腳本本身比傳輸腳本生成的命令開銷要小)。但是，這樣的做法帶
來的問題是 Redis 需要對腳本進(jìn)行一定約束：

The script always evaluates the same Redis write commands with the same
arguments given the same input data set. Operations performed by the scripts
cannot depend on any hidden information or state that may change as script
execution proceeds or between different exectuions of the script, nor can it
depend on any external input from I/O devices.

為了實現(xiàn)上述約束，Redis 對 Lua 運行環(huán)境和腳本的行為做了以下限制：

• Lua does not export commands to access the system time or other external state.

• Redis will block the script with an error if a script calls a Redis command
  able to alter the data set after a Redis random command like RANDOMKEY,
  SRANDMEMBER, TIME. This means that if a script is read-only and does not modify
  the data set it is free to call those commands. Note that a random command does
  not necessarily mean a command that uses random numbers: any non-deterministic
  command is considered a random command (the best example in this regard is the
  TIME command).

• Redis commands that may return elements in random order, like SMEMBERS
  (because Redis Sets are unordered) have a different behavior when called from
  Lua, and undergo a silent lexicographical sorting filter before returning data
  to Lua scripts. So redis.call(“smembers”,KEYS[1]) will always return the Set
  elements in the same order, while the same command invoked from normal clients
  may return different results even if the key contains exactly the same elements.

• Lua pseudo random number generation functions math.random and math.randomseed
  are modified in order to always have the same seed every time a new script is
  executed. This means that calling math.random will always generate the same
  sequence of numbers every time a script is executed if math.randomseed is not
  used.

• Redis scripts are not allowed to create global variables, in order to avoid
  leaking data into the Lua state. If a script needs to maintain state between
  calls (a pretty uncommon need) it should use Redis keys instead. In order to
  avoid using globals variables in your scripts simply declare every variable you
  are going to use using the local keyword.

同時，Redis 的 Lua 運行環(huán)境，提供了有限的模塊支持。同時，Redis 保證這些模塊在
各個 Redis 實例中都是一樣的。這樣就保證了腳本代碼在各個 Redis 實例中行為的一致
性。

Redis Lua 運行環(huán)境提供的模塊有: base, table, string, math, debug,
cjson, struct, cmsgpack。

Implementation

了解了 Redis 腳本相關(guān)操作和腳本限制后，再來分析一下 Redis 是如何實現(xiàn)上面提到的這
些特性的。

Lua API 的設(shè)用方法和與Lua交互的規(guī)范 (virtual stack 是如何使用的等)，可參閱
Programming in Lua；Lua API 的詳細(xì)說明，
請參閱 Lua API。

同時，需要注意的是，Redis 編譯時會鏈接自帶的 lua 代碼編譯出的靜態(tài)庫。同時，Redis
對 lua 源代碼進(jìn)行了擴(kuò)展，它將 cjson，struct，cmsgpack 變成了內(nèi)置模塊。

Lua 運行環(huán)境的初始化函數(shù)是 scriptingInit，在 Redis 啟動時，被 initServer
函數(shù)調(diào)用。

    ----------------scripting.c:527-----------------
    void scriptingInit(void) {
        lua_State *lua = lua_open();

        luaLoadLibraries(lua);
        luaRemoveUnsupportedFunctions(lua);
        ...
        lua_newtable(lua);

        /* redis.call */
        lua_pushstring(lua, "call");
        lua_pushcfunction(lua, luaRedisCallCommand);
        lua_settable(lua, -3);
        ...
        /* Finally set the table as 'redis' global var. */
        lua_setglobal(lua, "redis");

        /* Replace math.random and math.randomseed with our implementaions. */
        lua_getglobal(lua, "math");

        lua_pushstring(lua, "random");
        lua_pushcfunction(lua, redis_math_random);
        lua_settable(lua, -3);

        lua_pushstring(lua, "randomseed");
        lua_pushcfunction(lua, redis_math_randomseed);
        lua_settable(lua, -3);

        lua_setglobal(lua, "math");

        /* Add a helper function that we use to sort the multi bulk output
         * of non deterministic commands, when containing 'false' elements. */
        {
            char *compare_func =    "function __redis__compare_helper(a, b)\n"
                                    ...;
            luaL_loadbuffer(lua, compare_func, strlen(compare_func), "@cmp_func_def");
            lua_pcall(lua, 0, 0, 0);
        }

        /* Create the (non connected) client that we use to execute Redis commands
         * inside the Lua interpreter.
         * Note: there is no need to create it again when this function is called
         * by scriptingReset(). */
        if (server.lua_client == NULL) {
            server.lua_client = createClient(-1);
            server.lua_client->flags |= REDIS_LUA_CLIENT;
        }

        /* Lua beginners often don't use "local", this is likely to introduce
         * subtle bugs in their code. To prevent problems we protect accesses
         * to global variables. */
        scriptingEnableGlobalsProtection(lua);

        server.lua = lua;
    }

對上述代碼的補(bǔ)充說明：

• 關(guān)于 Lua API 的調(diào)用規(guī)范和細(xì)節(jié)說明，參閱上面的兩個鏈接。
• 上面代碼完成的工作有：
  • 創(chuàng)建一個新的 Lua 解釋器 (lua_State)
  • 加載類庫 base, table, string, math, debug, cjson, struct,
    cmsgpack。上文說過，cjson, struct 和 cmsgpack 是 Redis 添加到 lua
    核心代碼中的類庫。
  • 禁用可能帶來安全隱患的函數(shù)，比如 loadfile 等。
  • 在全局空間創(chuàng)建包含有 call, pcall, log 等 Redis 自定義函數(shù)的 table，
    并命名為 redis。這樣，在 EVAL 指令中的腳本就可以直接完成像 redis.call
    這樣的調(diào)用了。
  • 重新定義對 Redis 來講不安全的函數(shù) random 和 randomseed。
  • 在全局空間，定義函數(shù) __redis__compare_helper。
  • 創(chuàng)建 fake client，這樣腳本使用 Redis 命令時就能復(fù)用普通連接的命令執(zhí)行
    邏輯了。
  • 開啟”保護(hù)模式” – 禁止腳本聲明全局變量。

Redis 初始化完成后，開始監(jiān)聽客戶端請求。當(dāng)客戶端調(diào)用 EVAL, EVALSHA 等命令時，
Redis 才會調(diào)用腳本模塊進(jìn)行處理。

對客戶端請求的接收、命令解析等，本篇不作討論。下面只將函數(shù)調(diào)用鏈羅列如下：

    /* redis initialization */
    acceptTcpHandler
      anetTcpAccept
      acceptCommonHandler
        createClient
          readQueryFromClient
          /* wait read event */

    /* when data arrives */
    readQueryFromClient
      processInputBuffer
        processCommand
          lookupCommand
          call
            proc /* function pointer */

    /* for command `EVAL`, `proc` points to `evalCommand` */
    evalCommand
      evalGenericCommand

    /* for command `EVALSHA`, `proc` points to `evalShaCommand` */
    evalShaCommand
      evalGenericCommand

最后，evalGenericCommand 就是我們關(guān)心的在 lua 解釋器上執(zhí)行命令上傳的腳本的
入口函數(shù)。

    -----------scripting.c:787-----------------
    void evalGenericCommand(redisClient *c, int evalsha) {
        lua_State *lua = server.lua;
        ...
        funcname[0] = 'f';
        funcname[1] = '_';
        if (!evalsha) {
            /* Hash the code if this is an EVAL call */
            sha1hex(funcname + 2, c->argv[1]->ptr, sdslen(c->argv[1]->ptr));
        } else {
            /* We already have the SHA if it is a EVALSHA */
            ...
        }

        /* Try to lookup the Lua function */
        lua_getglobal(lua, funcname);
        if (lua_isnil(lua, 1)) {
            lua_pop(lua, 1); /* remove the nil from the stack */
            /* Function not defined... let's define it if we have the
             * body of the function. If this is an EVALSHA call we can just
             * return an error. */
            if (evalsha) {
                addReply(c, shared.noscripterr);
                return;
            }
            if (luaCreateFunction(c, lua, funcname, c->argv[1]) == REDIS_ERR) return;
            lua_getglobal(lua, funcname);
            redisAssert(!lua_isnil(lua, 1));
        }

        /* Populate the argv and keys table accordingly to the arguments that
         * EVAL received. */
        luaSetGlobalArray(lua, "KEYS", c->argv + 3, numkeys);
        luaSetGlobalArray(lua, "ARGV", c->argv + 3 + numkeys, c->argc - 3 - numkeys);
        ...
        /* At this point whatever this script was never seen before or if it was
         * already defined, we can call it. We have zero arguments and expect
         * a single return value. */
        err = lua_pcall(lua, 0, 1, 0);
        ...
        lua_gc(lua, LUA_GCSTEP, 1);

        if (err) {
            addReplyErrorFormat(c, "Error running script (call to %s): %s\n",
                funcname, lua_tostring(lua, -1));
            lua_pop(lua, 1); /* Consume the Lua reply. */
        } else {
            luaReplyToRedisReply(c, lua);
        }
        ...
    }

對上述代碼的補(bǔ)充說明：

• Redis 針對整個腳本字符計算 sha1。
• Redis 由客戶端上傳的腳本在 lua 全局作用域中創(chuàng)建 lua 函數(shù)，函數(shù)名為 f_sha1。
  此函數(shù)不接收參數(shù)，并且只能有一個返回值。
• KEYS 和 ARGV 的個數(shù)并不需要相等。Redis 根據(jù)其值在 lua 全局作用域中創(chuàng)建
  table。
• luaReplyToRedisReply 將 lua 函數(shù)的返回值，轉(zhuǎn)換成 Redis 類型。

Redis 腳本的處理和調(diào)用邏輯到此就算完成了。

如果腳本需要使用 Redis 命令 (大部分應(yīng)用場景都需要腳本和 Redis 進(jìn)行交互) 的話，就
需要使用 redis.call 和 redis.pcall 等 (還有 redis.log 等等的函數(shù)) 在初始化
環(huán)節(jié)注冊到 lua 環(huán)境中的函數(shù)。

下面以命令 EVAL "return redis.call('set', KEYS[1], ARGV[1])" 1 foo bar 為線索
對 Redis 提供的 redis.call 函數(shù)的執(zhí)行邏輯和如何從 lua 環(huán)境調(diào)用 C 函數(shù)進(jìn)行分析
(上面分析的過程，相當(dāng)于是在 C 函數(shù)中如何執(zhí)行 lua 代碼)。這樣一來，lua 和宿主語言
的兩個調(diào)用方向才算是完整了。

These two views of Lua (as an extension language and as an extensible language)
correspond to two kinds of interaction between C and Lua. In the first kind, C
has the control and Lua is the library. The C code in this kind of interaction
is what we call application code. In the second kind, Lua has the control and C
is the library. Here, the C code is called library code. Both application code
and library code use the same API to communicate with Lua, the so called C API.

在 lua 環(huán)境的初始化環(huán)節(jié)，Redis 將 luaRedisCallCommand 注冊為 lua 環(huán)境的
redis.call 函數(shù)。

    lua_pushstring(lua, "call");
    lua_pushcfunction(lua, luaRedisCallCommand);
    lua_settable(lua, -3);

luaRedisCallCommand 函數(shù)實現(xiàn)如下：

    -------------scripting.c:338-------------
    int luaRedisCallCommand(lua_State *lua) {
        return luaRedisGenericCommand(lua, 1);
    }

    -------------scripting.c:192-------------
    int luaRedisGenericCommand(lua_State *lua, int raise_error) {
        int j, argc = lua_gettop(lua);
        struct redisCommand *cmd;
        robj **argv;
        redisClient *c = server.lua_client;
        ...

        /* Build the arguments vector */
        argv = zmalloc(sizeof(robj *) * argc);
        for (j = 0; j < argc; j++) {
            if (!lua_isstring(lua, j+1)) break;
            argv[j] = createStringObject((char *) lua_tostring(lua, j + 1),
                                         lua_strlen(lua, j + 1);
        }
        ...
        /* Setup our fake client for command execution */
        c->argv = argv;
        c->argc = argc;

        /* Command lookup */
        cmd = lookupCommand(argv[0]->ptr);
        ...
        /* Run the command */
        c->cmd = cmd;
        call(c, REDIS_CALL_SLOWLOG | REDIS_CALL_STATS);
        ...
        redisProtocolToLuaType(lua, reply);
        ...
        return 1;
    }

對以上代碼的補(bǔ)充說明：

• Redis 接收到 EVAL 命令后，調(diào)用 evalGenericCommand 將腳本交給 Lua 解釋器執(zhí)
  行。Lua 解釋器執(zhí)行 return redis.call('set' KEYS[1], ARGV[1])。由于，redis.call
  又由 luaRedisCallCommand，這時，Lua 解釋器再調(diào)用此函數(shù)交命令交由 Redis 執(zhí)行。

• redis.call 中使用的命令 SET foo bar 由 fake client 作為載體執(zhí)行。

• 命令執(zhí)行結(jié)束后，Redis 將執(zhí)行結(jié)果使用 redisProtocolToLuaType 封裝成 Lua 類型
  并通過 Lua 調(diào)用協(xié)議寫入 Lua stack。

Redis 腳本的管理命令基本和 Lua 運行環(huán)境關(guān)系不大，在此就不再贅述了。