Atom Feed

myersguo's blog 2019-05-23T10:19:57+08:00 myersguo websocket, tcp 2019-04-04T00:00:00+08:00 /2019/04/04/tcp-websocket <p>some different object: i will write offently, every technology word will be an article.</p> <p>i will talk about: <a href="https://tools.ietf.org/html/rfc6455">websocket protocol</a>.</p> <p>websocket 协议包括两部分内容:</p> <ul> <li>handshake</li> <li>data transfer</li> </ul> <h4 id="handshake">handshake</h4> <p>通过 http 协议发送一个 协议升级的 chat:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>The handshake from the client looks as follows: </code></pre></div></div> <p>curl -v -i -H “Connection: Upgrade” -H “Upgrade: websocket” -H “Host: echo.websocket.org” -H “Origin: http://www.websocket.org” http://echo.websocket.org</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> GET /chat HTTP/1.1 Host: server.example.com Upgrade: websocket Connection: Upgrade Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ== Origin: http://example.com Sec-WebSocket-Protocol: chat, superchat Sec-WebSocket-Version: 13 The handshake from the server looks as follows: HTTP/1.1 101 Switching Protocols Upgrade: websocket Connection: Upgrade Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo= Sec-WebSocket-Protocol: chat ``` </code></pre></div></div> <p>一旦客户端和服务器端都发送了 handshakes 并成功连接, data transfer 开始。</p> <p>server code 101: <br /> HTTP/1.1 101 Switching Protocols<br /> 状态码: 101 表示 handshake 完成</p> <p>WebSocket 设计哲学是用最小化 framing.</p> <blockquote> <p>The WebSocket Protocol is an independent TCP-based protocol. Its only relationship to HTTP is that its handshake is interpreted by HTTP servers as an Upgrade request.</p> </blockquote> <p>By default, the WebSocket Protocol uses port 80 for regular WebSocket connections and port 443 for WebSocket connections tunneled over Transport Layer Security (TLS)</p> <h4 id="data-framing">Data Framing</h4> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-------+-+-------------+-------------------------------+ |F|R|R|R| opcode|M| Payload len | Extended payload length | |I|S|S|S| (4) |A| (7) | (16/64) | |N|V|V|V| |S| | (if payload len==126/127) | | |1|2|3| |K| | | +-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - + | Extended payload length continued, if payload len == 127 | + - - - - - - - - - - - - - - - +-------------------------------+ | |Masking-key, if MASK set to 1 | +-------------------------------+-------------------------------+ | Masking-key (continued) | Payload Data | +-------------------------------- - - - - - - - - - - - - - - - + : Payload Data continued ... : + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + | Payload Data continued ... | +---------------------------------------------------------------+ </code></pre></div></div> <p>看下发送的代码</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> def format(self): """ format this object to string(byte array) to send data to server. """ if any(x not in (0, 1) for x in [self.fin, self.rsv1, self.rsv2, self.rsv3]): raise ValueError("not 0 or 1") if self.opcode not in ABNF.OPCODES: raise ValueError("Invalid OPCODE") length = len(self.data) if length &gt;= ABNF.LENGTH_63: raise ValueError("data is too long") frame_header = chr(self.fin &lt;&lt; 7 | self.rsv1 &lt;&lt; 6 | self.rsv2 &lt;&lt; 5 | self.rsv3 &lt;&lt; 4 | self.opcode) if length &lt; ABNF.LENGTH_7: frame_header += chr(self.mask &lt;&lt; 7 | length) frame_header = six.b(frame_header) elif length &lt; ABNF.LENGTH_16: frame_header += chr(self.mask &lt;&lt; 7 | 0x7e) frame_header = six.b(frame_header) frame_header += struct.pack("!H", length) else: frame_header += chr(self.mask &lt;&lt; 7 | 0x7f) frame_header = six.b(frame_header) frame_header += struct.pack("!Q", length) if not self.mask: return frame_header + self.data else: mask_key = self.get_mask_key(4) return frame_header + self._get_masked(mask_key) </code></pre></div></div> <h4 id="tcp-server">tcp server</h4> <p>写一个 tcp server 来看一下, socket 通信过程</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># coding: utf-8 import socket import select import threading class TCPServer(): def __init__(self, host, port): self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.socket.bind((host, port)) self.socket.listen(10) def fileno(self): return self.socket.fileno() def serve_forever(self,): while True: r, w, e = select.select([self], [], [], 0.5) if self in r: self.handle_request() def handle_request(self, ): try: request_conn, client_address = self.socket.accept() except socket.error: return self.multi_process_request(request_conn, client_address) def multi_process_request(self, conn, addr): t = threading.Thread(target=self.process_request, args=(conn, addr)) t.daeamon = False t.start() def process_request(self, conn, addr): data = conn.recv(1024) if data: text = """ HTTP/1.1 302 Moved Temporarily Content-Type: text/html Location: https://www.google.com/ Proxy-Connection: Keep-alive """ print data conn.send(text) else: conn.close() return if __name__ == "__main__": server = TCPServer("0.0.0.0", 9008) server.serve_forever() </code></pre></div></div> event hook 2019-03-21T00:00:00+08:00 /2019/03/21/event-hook <p>event hook 是<a href="https://github.com/faif/python-patterns/blob/master/patterns/behavioral/observer.py">观察者模式</a>的典型应用。<code class="highlighter-rouge">provide a callback for notification of events/changes to data</code></p> <h3 id="django">django</h3> <p>用信号标识接收的方式,connect 表示接入, send 进行发送新号,所有接入方都将进行处理。</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class Signal(object): def __init__(self): self.receivers = [] def connect(self, receiver): self.receivers.append(receiver) def send(self, sender, **named): responses = [] for receiver in self.receivers: response = receiver(**named) responses.append((receiver, response)) return responses test_signal = Signal() def test(): print " go test" test_signal.connect(test) test_signal.send(sender="sender") </code></pre></div></div> <h3 id="flask">flask</h3> <p>flask 里的 signal 使用 <code class="highlighter-rouge">blinker</code> 的 <code class="highlighter-rouge">Signal</code></p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class Signal(object): def __init__(self): self.receivers = {} def connect(self, receiver): self.receivers.setdefault(receiver_id, receiver_ref) def send(self): return [(receiver, receiver(sender, **kwargs)) for receiver in self.receivers_for(sender)] </code></pre></div></div> <p>和 django 非常相似。</p> <h3 id="other">other</h3> <p>locust</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class EventHook(object): def __init__(self): self._handlers = [] def __iadd__(self, handler): self._handlers.append(handler) return self def fire(self, reverse=False, **kwargs): if reverse: self._handlers.reverse() for handler in self._handlers: handler(**kwargs) request_success = EventHook() def on_success(): print "on_success" request_success += on_success request_success.fire() </code></pre></div></div> <p>和 <code class="highlighter-rouge">django</code>, <code class="highlighter-rouge">flask</code> 类似。</p> decorator 2019-03-20T00:00:00+08:00 /2019/03/20/python-decorator <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def my_decorator(f): def wrapper(): print "before" f() print "after" return wrapper def test(): print "test" my_test = my_decorator(test) my_test() print test.__name__ print my_test.__name__ </code></pre></div></div> <p>每个函数作为「注解」的参数,注解的调用返回一个 wrapper 函数。 这个时候,my_test 变成了 <code class="highlighter-rouge">wrapper</code></p> <h3 id="语法糖-syntactic-sugar">@语法糖 syntactic sugar</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@my_decorator def test2(): print "test2" #test2 = my_decorator(test2) test2() </code></pre></div></div> <p>函数参数:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def my_decorator(f): def wrapper(*args, **kwargs): print "before" f(*args, **kwargs) print "after" return wrapper def test(name): print "test", name @my_decorator def test2(name): print "test2", name my_test = my_decorator(test) my_test("xiaoming") test2("xiaoming2") </code></pre></div></div> <p>注解带参数:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def with_params_decorator(n=0): def my_decorator(f): @functools.wraps(f) def wrapper(*args, **kwargs): print n print "before" f(*args, **kwargs) print "after" return wrapper return my_decorator </code></pre></div></div> <p>多套了一层。。。</p> <p>为了解决, python 类型自省变动问题,使用 <code class="highlighter-rouge">@functools.wrapper</code>来解决。来看下这个注解:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> WRAPPER_ASSIGNMENTS = ('__module__', '__name__', '__doc__') WRAPPER_UPDATES = ('__dict__',) def update_wrapper(wrapper, wrapped, assigned = WRAPPER_ASSIGNMENTS, updated = WRAPPER_UPDATES): """Update a wrapper function to look like the wrapped function wrapper is the function to be updated wrapped is the original function assigned is a tuple naming the attributes assigned directly from the wrapped function to the wrapper function (defaults to functools.WRAPPER_ASSIGNMENTS) updated is a tuple naming the attributes of the wrapper that are updated with the corresponding attribute from the wrapped function (defaults to functools.WRAPPER_UPDATES) """ for attr in assigned: setattr(wrapper, attr, getattr(wrapped, attr)) for attr in updated: getattr(wrapper, attr).update(getattr(wrapped, attr, {})) # Return the wrapper so this can be used as a decorator via partial() return wrapper def wraps(wrapped, assigned = WRAPPER_ASSIGNMENTS, updated = WRAPPER_UPDATES): """Decorator factory to apply update_wrapper() to a wrapper function Returns a decorator that invokes update_wrapper() with the decorated function as the wrapper argument and the arguments to wraps() as the remaining arguments. Default arguments are as for update_wrapper(). This is a convenience function to simplify applying partial() to update_wrapper(). """ return partial(update_wrapper, wrapped=wrapped, assigned=assigned, updated=updated) </code></pre></div></div> <p>wraps 这个返回的是 partial 更新了的 <code class="highlighter-rouge">update_wrapper</code>,默认值 wrapped=wrapperd,即:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def wraps(wrapped): def new_wraps(): return update_wrapper(wrapped=wrappwd) return new_wraps </code></pre></div></div> <p>被 functions.wraps 注解的函数,的 name, doc 等都不会变。</p> <h3 id="类注解">类注解</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class class_decorator(object): def __init__(self, f): functools.update_wrapper(self, func) self.f = f def __call__(self, *args, **kwargs): print "before" self.f(*args, **kwargs) print "after" my_test = class_decorator(test) my_test("class") </code></pre></div></div> <p>类注解满足两点:1) <strong>init</strong> 传入 func, 2) <strong>call</strong> 可执行</p> mysql index 2019-03-18T00:00:00+08:00 /2019/03/18/mysql-index <p>问题:一个 student 表,有100 万条记录,假如 status = 0 的有90万,剩下的 status &gt;0 的10 万,每个人都有一个属性 school, 现在要查询, school 为空的, status&gt;0 的数据。</p> <p><code class="highlighter-rouge">select count(id) from student where school = '' and status</code><br /> 我们发现,mysql 使用 school 作为索引,没有使用 status,这就造成了需要从 90 万查找数据。如果使用 status 则只需要从 10 万种查找数据了。怎么优化,如果这是一种常态,可以使用嵌套查询.</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>select count(id) from (select id, status) from student where status&gt;0) as c where school = ''; </code></pre></div></div> <p>或者使用 force index<code class="highlighter-rouge">FORCE INDEX('idx_student_status')</code><br /> ```</p> <h3 id="索引的结构">索引的结构</h3> <p>索引内容包含两部分。 索引值 + 块地址。</p> <p><code class="highlighter-rouge"> the clustered index is synonymous with the primary key. </code> 主键 key 就是聚集索引。</p> <p><img src="https://www.geeksforgeeks.org/wp-content/uploads/gq/2016/07/indexing3.png" alt="https://www.geeksforgeeks.org/wp-content/uploads/gq/2016/07/indexing3.png" /></p> <h3 id="参考">参考</h3> <p><a href="https://dev.mysql.com/doc/refman/5.7/en/innodb-index-types.html">https://dev.mysql.com/doc/refman/5.7/en/innodb-index-types.html</a> <br /> <a href="https://github.com/jeremycole/innodb_diagrams/blob/master/images/InnoDB_Structures.pdf">https://github.com/jeremycole/innodb_diagrams/blob/master/images/InnoDB_Structures.pdf</a></p> <p><a href="https://www.geeksforgeeks.org/indexing-in-databases-set-1/">https://www.geeksforgeeks.org/indexing-in-databases-set-1/</a> <br /> <a href="https://monkeysayhi.github.io/2018/03/06/%E6%B5%85%E8%B0%88MySQL%E7%9A%84B%E6%A0%91%E7%B4%A2%E5%BC%95%E4%B8%8E%E7%B4%A2%E5%BC%95%E4%BC%98%E5%8C%96/">https://monkeysayhi.github.io/2018/03/06/%E6%B5%85%E8%B0%88MySQL%E7%9A%84B%E6%A0%91%E7%B4%A2%E5%BC%95%E4%B8%8E%E7%B4%A2%E5%BC%95%E4%BC%98%E5%8C%96/</a><br /> <a href="https://blog.jcole.us/innodb/">https://blog.jcole.us/innodb/</a> <br /> <a href="https://www.cnblogs.com/leefreeman/p/8315844.html">https://www.cnblogs.com/leefreeman/p/8315844.html</a></p> what is thread local 2019-03-17T00:00:00+08:00 /2019/03/17/thread-local <p>进程之间是不共享内存的。不存在数据争强的问题,但是线程的数据是同内存种的,多线程模式下,对同一变量进行修改,可能造成数据不一致的情况。有两种解决办法:使用局部变量或者thread local variable。<br /> 使用局部变量有一个问题是在多个函数共同使用要通过参数传递方式实现。</p> <p>线程争强例子:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import threading total_cnt = 0 def process_1(): global total_cnt for x in xrange(500): total_cnt += 1 if __name__ == "__main__": for x in xrange(2): t = threading.Thread(target=process_1) t.start() print total_cnt </code></pre></div></div> <p>执行100次上面的代码: <br /> <code class="highlighter-rouge">for((x=0;x&lt;100;x+=1));do python thread_local.py;done</code> 你会发现结果不是全部都是 1000。 因为多线程下global variable 不是原子操作。对同一个数据操作需要加锁。</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>l = threading.Lock() def process_1(): global total_cnt if l.acquire(): try: for x in xrange(500): total_cnt += 1 finally: l.release() </code></pre></div></div> <p>local thread 一般用于解决线程之间数据隔离的问题。假如有一个 global 的数据,如果只在当前线程下有效,就可以使用 threading.local。 它是怎么实现的呢?可以参考<a href="https://github.com/pallets/werkzeug/blob/master/src/werkzeug/local.py">werkzeug</a>的实现。它使用<code class="highlighter-rouge">__storage__</code>来保存变量,每个线程使用不同的dict 的 id (通过 get_ident) 区分, 从而保证了线程的安全。</p> design pattern 2019-03-17T00:00:00+08:00 /2019/03/17/python-design-pattern <h3 id="singleton-pattern-单例模式">Singleton Pattern 单例模式</h3> <p>import once, so singleton as following:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class Singleton(object): def test(): pass singleton_service = Singleton() #我们最常用的方式 </code></pre></div></div> <p>or:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class Singleton(object): _instance = None def __new__(class_, *args, **kwargs): if not isinstance(class_._instance, class_): class_ = super(Singleton, class_).__new__(class_, *args, **kwargs) return class_._instance </code></pre></div></div> <p>or metaclass</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class Singleton(type): _instances = {} def __call__(cls, *args, **kwargs): if cls not in cls._instances: cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs) return cls._instances[cls] class Logger(object): __metaclass__ = Singleto </code></pre></div></div> <h3 id="borg-pattern">borg pattern</h3> <p>将<code class="highlighter-rouge">__dict__</code> 绑定到全局对象上。 造成单例的功能。</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class Borg(object): __shared_state = {} def __init__(self): self.__dict__ = self.__shared_state self.state = 'Init' def __str__(self): return self.state </code></pre></div></div> <h3 id="adapter-适配器模式">adapter 适配器模式</h3> <blockquote> <p>the adapter pattern is a software design pattern (also known as wrapper, an alternative naming shared with the decorator pattern) that allows the interface of an existing class to be used as another interface. It is often used to make existing classes work with others without modifying their source code.</p> </blockquote> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import functools class AdapterClient(object): def __init__(self, **kwargs): pass def __getattr__(self, item): def wrapper(self, *args, **kwargs): print "run:", item return functools.partial(wrapper, self) class Adapter(object): def __init__(self, **kwargs): self.client = AdapterClient() def __getattr__(self, item): return getattr(self.client, item) adpt = Adapter(a=1) adpt.add() adpt.minus() </code></pre></div></div> <h3 id="factory-pattern-工厂模式">factory pattern 工厂模式</h3> <p>In class-based programming, the factory method pattern is a creational pattern that uses factory methods to deal with the problem of creating objects without having to specify the exact class of the object that will be created. This is done by creating objects by calling a factory method—either specified in an interface and implemented by child classes, or implemented in a base class and optionally overridden by derived classes—rather than by calling a constructor.</p> <p>创建不同的对象。</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class AFactory(object): def get_obj(self, Obj): return Obj() class Cat(): def test(self): print "cat" class Dog(): def test(self): print "dog" a = AFactory() cat = a.get_obj(Cat) cat.test() dog = a.get_obj(Dog) dog.test() </code></pre></div></div> <h3 id="参考">参考</h3> <p><a href="https://github.com/faif/python-patterns">https://github.com/faif/python-patterns</a></p> Locust 2019-03-17T00:00:00+08:00 /2019/03/17/locust <h3 id="keywords">keywords</h3> <p>locust(用户): 代表了一个 “user” 用于发起请求。它的行为定义在任务集合<code class="highlighter-rouge">task_set</code>中(定义了任务的执行集合)。其他一些配置:任务之间的等待时间,超时设置。 taskset(任务场景): 包含要执行的任务集合。 run 即从执行任务里面随机选择一个task 到 执行队列(schedule_task) 然后 execute_task, TaskSequence(顺序任务集合): 上面的任务场景是随机执行 tasks,而这个是按照顺序执行 tasks。 LocustRunner(执行者):<br /> 执行用户的 runner/agent。可以执行(start_hatching)。分布式执行需要 MasterLocustRunner, 和 slave runner&gt;</p> <h3 id="step-by-step">step by step</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/locustio/locust.git cd locust/example # master 启动,10个用户 qps 100 locust -f basic.py --master --no-web -c 10 -r 100 --run-time 1m --loglevel=DEBUG # slave 启动 locust -f basic.py --slave --loglevel=DEBUG </code></pre></div></div> <p>result:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[2019-05-19 17:38:55,233] myersguo/INFO/root: Waiting for slaves to be ready, 0 of 1 connected [2019-05-19 17:39:08,690] myersguo/INFO/locust.runners: Client u'myersguo_1aa023183b584394bfc4b5b946257d13' reported as ready. Currently 1 clients ready to swarm. [2019-05-19 17:39:09,277] myersguo/INFO/locust.runners: Sending hatch jobs to 1 ready clients [2019-05-19 17:39:09,278] myersguo/INFO/locust.main: Run time limit set to 60 seconds [2019-05-19 17:39:09,278] myersguo/INFO/locust.main: Starting Locust 0.11.1 Name # reqs # fails Avg Min Max | Median req/s -------------------------------------------------------------------------------------------------------------------------------------------- GET / 19 19(50.00%) 1 8 49 | 10 0.90 GET /does_not_exist 23 23(50.00%) 12 7 50 | 10 1.10 GET /stats/requests 10 10(50.00%) 10 8 13 | 10 0.70 -------------------------------------------------------------------------------------------------------------------------------------------- Total 52 52(100.00%) </code></pre></div></div> <h3 id="keywords-others">keywords others</h3> <p>对比一下 loadrunner 的技术。</p> <ul> <li>Scenario: 场景。包含一列些 events,来模拟用户的真实行为。</li> <li>Vusers(Protocol Based Vusers): 在场景中,lr使用 virtual users 来表示用户。</li> <li>Vuser scripts: vuser 在场景中执行的脚本。包含各种函数。</li> <li>Transactions: 衡量服务端的压力的纬度。代表了行为集合。</li> <li>Rendezvous points: 集合点。用于多用户等待集合。</li> <li>Controller: 控制器。 用于管理场景。</li> <li>Load Generator: agent 用户执行 vuser 脚本。</li> <li>Performance analysis. 性能统计分析。</li> </ul> <p>我们看到 user, scenario(task test), script, agent, stats 这几个是基本的概念。</p> thrift 2019-03-14T00:00:00+08:00 /2019/03/14/thrift <p>先走一遍官网的例子:</p> <p>用到的两个文件<a href="https://github.com/apache/thrift/blob/master/tutorial/tutorial.thrift">tutorial.thrift</a> and <a href="https://github.com/apache/thrift/blob/master/tutorial/shared.thrift">shared.thrift</a>.</p> <p><code class="highlighter-rouge"> ~/thrift/bin/thrift -r --gen py:utf8strings -out ./thrift_gen ./tutorial.thrift </code></p> <p>生成三个文件: constants,ttypes,Calculator.py &amp;&amp; SharedService.py</p> <p>其中:constants 定义的是 const 变量,thrift 即里面的const 会放到这里:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const i32 INT32CONSTANT = 9853 const map&lt;string,string&gt; MAPCONSTANT = {'hello':'world', 'goodnight':'moon'} </code></pre></div></div> <p>ttypes 里面定义 enum 类型和 struct 类型:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class Operation: ADD = 1 SUBTRACT = 2 MULTIPLY = 3 DIVIDE = 4 </code></pre></div></div> <p>struct 会定义好 read,write 的方法:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class Work: def __init__(...): pass def read(): ... def write(): ... </code></pre></div></div> <p>最后就是 service file ,里面定义暴露的接口:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def getStruct(self, key): self.send_getStruct(key) return self.recv_getStruct() </code></pre></div></div> <p><code class="highlighter-rouge">TProcessor</code>,<code class="highlighter-rouge">TTransport</code>,<code class="highlighter-rouge">TBinaryProtocol</code>,<code class="highlighter-rouge">TProtocol</code>,<code class="highlighter-rouge">fastbinary</code></p> <p>TProtocol 协议, TTransport: socket; TProcessor 方法处理;</p> <p>TProtocol 定义通信的协议。thrift TProtocolBase 协议定义了协议的开始、结束方法。writeMessageBegin 定义消息头,writeStructBegin 定义消息方法,writeFieldBegin 定义数据类型。eg:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> def writeMessageBegin(self, name, type, seqid): assert self.state == CLEAR self.__writeUByte(self.PROTOCOL_ID) self.__writeUByte(self.VERSION | (type &lt;&lt; self.TYPE_SHIFT_AMOUNT)) self.__writeVarint(seqid) self.__writeString(name) self.state = VALUE_WRITE def writeMessageEnd(self): assert self.state == VALUE_WRITE self.state = CLEAR def writeStructBegin(self, name): assert self.state in (CLEAR, CONTAINER_WRITE, VALUE_WRITE), self.state self.__structs.append((self.state, self.__last_fid)) self.state = FIELD_WRITE self.__last_fid = 0 </code></pre></div></div> <p>结合 <a href="https://gitbox.apache.org/repos/asf?p=thrift.git;a=blob_plain;f=tutorial/py/PythonClient.py;hb=HEAD">client</a>和<a href="https://gitbox.apache.org/repos/asf?p=thrift.git;a=blob;f=tutorial/py/PythonServer.py;hb=HEAD">server</a>的代码我们理解一下:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Make socket, 创建 socket, transport from thrift.transport import TSocket transport = TSocket.TSocket('localhost', 9090) # Buffering is critical. Raw sockets are very slow, from thrift.transport import TTransport transport = TTransport.TBufferedTransport(transport) # Wrap in a protocol, 协议初始化 protocol = TBinaryProtocol.TBinaryProtocol(transport) # Create a client to use the protocol encoder, 客户端 with proto client = Calculator.Client(protocol) # Connect! transport.open() client.ping() print('ping()') </code></pre></div></div> <p>processor 是解析、处理类,processMap是所有方法的 map。 每个 process 的过程是:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(name, type, seqid) = iprot.readMessageBegin() args.read() iprot.readMessageEnd() initResp hanle it: send recv write flush </code></pre></div></div> <p>transport 的过程:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> def read(self, sz): ret = self.__rbuf.read(sz) if len(ret) != 0: return ret self.__rbuf = StringIO(self.__trans.read(max(sz, self.__rbuf_size))) return self.__rbuf.read(sz) </code></pre></div></div> <p>TSocket:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import socket def open(self): try: res0 = self._resolveAddr() for res in res0: self.handle = socket.socket(res[0], res[1]) self.handle.settimeout(self._timeout) try: self.handle.connect(res[4]) except socket.error, e: if res is not res0[-1]: continue else: raise e break except socket.error, e: if self._unix_socket: message = 'Could not connect to socket %s' % self._unix_socket else: message = 'Could not connect to %s:%d' % (self.host, self.port) raise TTransportException(type=TTransportException.NOT_OPEN, message=message) </code></pre></div></div> <p>画重点,<strong>transport</strong> 就是 <strong><a href="https://docs.python.org/2/library/socket.html">socket</a></strong> 的传输。</p> <p>Client:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>send: self._oprot.writeMessageBegin args init self._oprot.writeMessageEnd flush recv: (fname, mtype, rseqid) = iprot.readMessageBegin() result.read(iprot) iprot.readMessageEnd() </code></pre></div></div> <p>看下 server 的过程:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class TSimpleServer(TServer): """Simple single-threaded server that just pumps around one transport.""" def __init__(self, *args): TServer.__init__(self, *args) def serve(self): self.serverTransport.listen() while True: client = self.serverTransport.accept() itrans = self.inputTransportFactory.getTransport(client) otrans = self.outputTransportFactory.getTransport(client) iprot = self.inputProtocolFactory.getProtocol(itrans) oprot = self.outputProtocolFactory.getProtocol(otrans) try: while True: self.processor.process(iprot, oprot) except TTransport.TTransportException, tx: pass except Exception, x: logging.exception(x) itrans.close() otrans.close() </code></pre></div></div> <p>process 一直跑….</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import socket sk = socket.socket() sk.bind(("127.0.0.1", 9092)) sk.listen(5) while True: client, addr = sk.accept() while True: data = client.recv(1024) if not data: break print "get from client:%s" % data client.sendall(data) client.close() client.py import socket s = socket.socket() s.connect(("127.0.0.1", 9092)) s.sendall("i am client") print "get from server:%s" % s.recv(1024) s.close() </code></pre></div></div> <p>整个流程下来就是:</p> <p>client 端: 建立socket 连接, 按照 proto 从 socket 里读数据。server 端:建立 socket,listen 连接, process 请求。</p> gevent 2019-03-08T00:00:00+08:00 /2019/03/08/gevent <h3 id="greenlet">greenlet</h3> <h3 id="hub--loop"><a href="http://www.gevent.org/api/gevent.hub.html">hub &amp;&amp; loop</a></h3> <p><a href="http://www.gevent.org/api/gevent.hub.html#the-event-loop">event loop</a>, hub 是一个单例的 greenlet, 用来调度 greenlet,每个 greenlet 都有一个 parent greenlet, loop server 在 hub里循环。<code class="highlighter-rouge">hub = get_hub(); loop = hub.loop;</code></p> <p>未完待续..</p> <h3 id="patch">patch</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def patch_item(module, attr, newitem): NONE = object() olditem = getattr(module, attr, NONE) if olditem is not NONE: saved.setdefault(module.__name__, {}).setdefault(attr, olditem) setattr(module, attr, newitem) def patch_module(name, items=None): gevent_module = getattr(__import__('gevent.' + name), name) module_name = getattr(gevent_module, '__target__', name) module = __import__(module_name) if items is None: items = getattr(gevent_module, '__implements__', None) if items is None: raise AttributeError('%r does not have __implements__' % gevent_module) for attr in items: patch_item(module, attr, getattr(gevent_module, attr)) </code></pre></div></div> <p>以上是 gevent 的 patch module 的过程。现在简化一下,方便理解:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># 当前目录有: #f1.py: def test(): print "test in current module" #modules/f1.py def test(): print "test in modules" #patch.py def patch_module(name): my_module = getattr(__import__('modules.' + name), name)#modules.f1 module_name = getattr(my_module, '__target__', name)#f1 module = __import__(module_name) #module f1 in current items = dir(my_module) for attr in items: setattr(module, attr, getattr(my_module, attr)) patch_module("f1") import f1 f1.test() </code></pre></div></div> <blockquote> <p>using <strong>import</strong> can be that it returns the imported module and doesn’t add anything to the namespace; you can import with it without having then to delete the new name if you didn’t want that new name; using import somename will add somename to your namespace, but <strong>import</strong>(‘somename’) instead returns the imported module, which you can then ignore.</p> </blockquote> <h3 id="geventthread">gevent.thread</h3> <p>patch 了以下方法:</p> <p>get_ident: 更改为获取当前协程的地址 <code class="highlighter-rouge">id(greenlet.getcurrent())</code></p> <p>start_new_thread:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def start_new_thread(function, args=(), kwargs={}): greenlet = Greenlet.spawn(function, *args, **kwargs) return get_ident(greenlet) </code></pre></div></div> 个人 OKR 实践 2019-02-22T00:00:00+08:00 /2019/02/22/my-okr <p>从 2.10 号开始,我开始尝试一个自我管理办法:用双周 OKR 来管理自己的目标。OKR 这个词最初是在小米使用,当时感觉像是在写周报一样,管理层也没有把 OKR 推广出去。来到头条后,发现每个人都要写OKR,从最上级到最底层得员工,OKR 向上对齐,完成公司的指定目标。</p> <p>某一天,我看到「凡事皆项目」,就知道我应该把自己也规划一下了。,然后写了以下 OKR:</p> <h4 id="210-224-project-g-build">2.10-2.24 project G build</h4> <ul> <li>读书与写作:读书习惯的初步建立 0.5 分(mysql 的书完成了30%, 中间时间因为某些原因断了) <ul> <li>上班路上养成阅读习惯</li> <li>完成 mysql 技术内部的阅读、redis 实践阅读1章</li> <li>每天下班路上完成3篇 micro notes</li> <li>完成2篇文章:一篇读书笔记、一篇思考</li> </ul> </li> <li>跑步:跑步习惯初步养成 0.8分 每周五坚持跑步了1小时,早起没有养成 <ul> <li>完成一次 10 公里</li> <li>每天6点30 准时起床</li> <li>每周一次跑步</li> </ul> </li> <li>工作:高效、高质量完成工作 0.5 分 ticket system 完成,mysql search 没有开展 <ul> <li>ticket system 完成提测、单元测试覆盖率 80% 以上</li> <li>初步完成 mysql 检索调研,完成一次笔记分享</li> </ul> </li> </ul> <p>事实证明, 规划自己的 OKR 是有效的。可以约束自己的行动,做到心中有数。</p> <p>现在互联网寒冬来了,而且,随着一个人的核心价值随着时间的推移而下降,不能通过规划来成长,就会逐渐被淘汰。</p>