Atom Feed

myersguo's blog 2018-08-18T21:34:50+08:00 myersguo algorithm 2018-08-13T00:00:00+08:00 /2018/08/13/algorithm <h3 id="用两个堆栈模拟队列的功能-pushpopcount">用两个堆栈模拟队列的功能: push\pop\count</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def stack_push(a, b, item): """ 入队操作,判断 a 是否为空,为空则将b中数据压入a,不为空,则直接入栈 """ if len(a) &lt;= 0: while len(b) &gt; 0: a.append(b.pop()) a.append(item) def stack_pop(a, b): """ 出队操作:判断 b 是否为空,不为空直接弹出;为空则将a压入b中 """ if len(b) &gt;0: return b.pop() while len(a) &gt; 1: b.append(a.pop()) if len(a) &gt; 0: return a.pop() def stack_count(a, b): return len(a) if len(b) &lt;=0 else len(b) a, b = [], [] if __name__ == "__main__": stack_pop(a, b) stack_push(a, b, 1) stack_push(a, b, 3) stack_push(a, b, 5) assert stack_count(a, b) == 3 assert stack_pop(a, b) == 1 assert stack_pop(a, b) == 3 assert stack_count(a, b) == 1 stack_push(a, b, 4) assert stack_count(a, b) == 2 assert stack_pop(a, b) == 5 assert stack_pop(a, b) == 4 </code></pre></div></div> <h3 id="二叉树最长路径">二叉树最长路径</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># 二叉树节点 class Node(object): def __init__(self, elem=None, lChild=None, rChild=None): self.elem = elem self.lChild = lChild self.rChild = rChild self.parent = None def __str__(self): if self.elem: return str(self.elem) # 二叉树最长路径 def find_tree_max_path(root): """ 层次遍历二叉树,对每个节点进行标记,标记其父节点 """ treeList = [] treeList.append(root) _tmp, index = treeList[0], 0 while True: if _tmp.lChild is not None: treeList.append(_tmp.lChild) print 'add new item', _tmp.lChild _tmp.lChild.parent = _tmp if _tmp.rChild is not None: treeList.append(_tmp.rChild) print 'add new item', _tmp.rChild _tmp.rChild.parent = _tmp index +=1 if index &gt;= len(treeList): break _tmp = treeList[index] # 输出最长路径 item = treeList[len(treeList)-1] while item: print item item = item.parent if __name__ == "__main__": a, b, c, d, e, f, g = Node(1), Node(2), Node(3),Node(4),Node(5),Node(6),Node(7) a.lChild, a.rChild = b, c b.lChild, b.rChild = d, e e.lChild, e.rChild = f, g find_tree_max_path(a) </code></pre></div></div> <h3 id="android-解锁密码">android 解锁密码</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>from itertools import permutations, chain impossible = {'91': '5', '13': '2', '17': '4', '46': '5', '37': '5', '31': '2', '28': '5', '39': '6', '19': '5', '64': '5', '97': '8', '71': '4', '82': '5', '73': '5', '93': '6', '79': '8'} # 生成所有的组合类型 allIterate = chain(*(permutations('123456789', i) for i in range(4, 10))) count = 0 for item in allIterate: stri = ''.join(item) for k, v in impossible.items(): if k in stri and v not in stri[:stri.find(k)]: break else: count += 1 </code></pre></div></div> <h3 id="出现此处大于-m-的-数组元素">出现此处大于 m 的 数组元素</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>a = [1, 1, 3, 1, 2, 3, 1, 2, 3, 4, 5, 4, 5, 4, 1] def findArr(a, m=3): result = [] hashSet = {} for item in a: if hashSet.get(item): hashSet[item] +=1 else: hashSet[item] = 1 for item in hashSet: if hashSet[item] &gt; m: result.append(item) return result print findArr(a, 4) </code></pre></div></div> <p>假如我们期望 hashSet 的空间尽量小,则:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def findArr2(a, m=3): result, hashSet = [], {} lenA = len(a) lenH = lenA / m for item in a: if hashSet.get(item) is not None: if hashSet[item] &lt;=0: del hashSet[item] continue if len(hashSet) &gt; lenH: for _item in hashSet: hashSet[_item] -=1 hashSet[item] += 1 else: hashSet[item] = 1 for item in hashSet: if hashSet[item] &gt; m: result.append(item) return result </code></pre></div></div> <h3 id="有两个字典分别存有-100-条数据和-10000-条数据如果用一个不存在的-key-去查找数据在哪个字典中速度更快">有两个字典,分别存有 100 条数据和 10000 条数据,如果用一个不存在的 key 去查找数据,在哪个字典中速度更快?</h3> <p>hash 表的满的程度用负载因子体现。负载因子越大,意味着哈希表越满,越容易导致冲突,性能也就越低。因此,一般来说,当负载因子大于某个常数(可能是 1,或者 0.75 等)时,哈希表将自动扩容。<a href="https://bestswifter.com/hashtable/">参考</a>。hash 扩容 rehash 过程一般2倍空间的申请,将原来的数据复制到新的空间。</p> <p>redis的扩容,详见:<a href="http://redisbook.com/preview/dict/incremental_rehashing.html">渐进式hash</a></p> consul 2018-07-26T00:00:00+08:00 /2018/07/26/consul <h3 id="概念">概念</h3> <p>nodes: 实例节点 <br /> services: 服务(一般是 http/rpc client)</p> <h3 id="install">install</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># download from :https://www.consul.io/downloads.html # start agent consul agent -dev # test consul members curl localhost:8500/v1/catalog/nodes # tag service echo '{"service": {"name": "web", "tags": ["rails"], "port": 80}}' \ | sudo tee /etc/consul.d/web.json </code></pre></div></div> <p>test dns:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dig @127.0.0.1 -p 8600 web.service.consul web.service.consul. 0 IN A 127.0.0.1 </code></pre></div></div> <h3 id="http-api">HTTP API</h3> <p>set key v</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl --request PUT --data bar http://localhost:8500/v1/kv/foo </code></pre></div></div> <p>get key v</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl http://localhost:8500/v1/kv/foo </code></pre></div></div> <p>delete key v</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl --request DELETE http://localhost:8500/v1/kv/foo </code></pre></div></div> <h4 id="agent">agent</h4> <p>agent members:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl http://localhost:8500/v1/agent/members </code></pre></div></div> <p>reload</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl --request PUT http://localhost:8500/v1/agent/reload </code></pre></div></div> <p>log monitor:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>http://localhost:8500/v1/agent/monitor?loglevel=WARN </code></pre></div></div> <h4 id="event">event</h4> <p>show</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl http://localhost:8500/v1/event/list </code></pre></div></div> <h4 id="catalog">catalog</h4> <p>list node:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl http://localhost:8500/v1/catalog/nodes </code></pre></div></div> <p>dc list:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl http://localhost:8500/v1/catalog/datacenters </code></pre></div></div> <p>services:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl http://localhost:8500/v1/catalog/services </code></pre></div></div> <p><em>List Nodes for Service</em>:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl http://localhost:8500/v1/catalog/services/myersguo.github.com </code></pre></div></div> <p><em>List Service for node</em>:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl http://10.8.163.168:2280/v1/catalog/node/n6-028-xxx </code></pre></div></div> flask 2018-07-16T00:00:00+08:00 /2018/07/16/about-flask <p>flask 的 uwsgi 是基于<a href="http://werkzeug.pocoo.org/">werkzeug</a>, django 是自己实现了一套 wsgi 协议。</p> <h3 id="路由解析">路由解析</h3> <p>flask 的路由由 werkzeug 的 routing 模块实现,我们来看 flask 的封装:</p> <p>入口 <code class="highlighter-rouge">wsgi_app</code> :</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ctx = self.request_context(environ) # return RequestContext(self, environ) ctx.push() error = None try: try: response = self.full_dispatch_request() except Exception as e: error = e response = self.make_response(self.handle_exception(e)) return response(environ, start_response) finally: if self.should_ignore_error(error): error = None ctx.auto_pop(error) </code></pre></div></div> <h4 id="dispatch_request">dispatch_request</h4> <p>`return self.view_functions<a href="**req.view_args">rule.endpoint</a><br /> 这里的 view_functions 是在添加规则时(add_url_rule)设置的, 路由 rule 与 对应的 view 关系 list。</p> <h4 id="url_adapter-url_map">url_adapter, url_map</h4> <p>werkzeug.routing:</p> <p>create_url_adapter:创建一个 url dapater<br /> url_adapter.match: 获取匹配的 url rule<br /> self.view_functions: <a href="**req.view_args">rule.endpoint</a>: 执行对应的 function</p> <p>app.add_url_rule(): 添加路由,相当于注解 app.route:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def index(): pass app.add_url_rule('/', 'index', index) 相当于 @app.route('/') def index(): pass </code></pre></div></div> <p>rule 包括三个部分: converter, arguments, variable<br /> converter(arguments):name&gt;<br /> 比如:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import os from werkzeug.wrappers import Request, Response from werkzeug.wsgi import SharedDataMiddleware from werkzeug.routing import Map, Rule from werkzeug.exceptions import HTTPException, NotFound from werkzeug.routing import BaseConverter class RegexConverter(BaseConverter): def __init__(self, url_map, *args): super(RegexConverter, self).__init__(url_map) self.regex = args[0] class Shortly(object): def __init__(self): self.url_map = Map([ Rule('/', endpoint='new_url'), Rule('/&lt;regex(".*"):others&gt;', endpoint='new_url') ], converters={'regex': RegexConverter}) def dispatch_request(self, request): return Response('Hello World!') def dispatch_request(self, request): adapter = self.url_map.bind_to_environ(request.environ) try: endpoint, values = adapter.match() return getattr(self, 'on_' + endpoint)(request, **values) except HTTPException, e: return e def on_new_url(self, request, others=None): return Response('Hello World!') def wsgi_app(self, environ, start_response): request = Request(environ) response = self.dispatch_request(request) return response(environ, start_response) def __call__(self, environ, start_response): return self. wsgi_app(environ, start_response) def create_app(redis_host='localhost', redis_port=6379, with_static=True): app = Shortly() if with_static: app.wsgi_app = SharedDataMiddleware(app.wsgi_app, { '/static': os.path.join(os.path.dirname(__file__), 'static') }) return app if __name__ == '__main__': from werkzeug.serving import run_simple app = create_app() run_simple('127.0.0.1', 5000, app, use_debugger=True, use_reloader=True) </code></pre></div></div> <h3 id="middleware">middleware</h3> <p>app.wsgi_app = MyMiddleware(app.wsgi_app)</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>flask 的中间件是通过一些阶段方法来实现(before, after, tear down) </code></pre></div></div> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>before_request: app.before_request(f) after_request: app.after_request(f) teardown_request: app.teardown_request(f) </code></pre></div></div> <h3 id="ctxflaskctx-请求上下文">ctx(flask.ctx) 请求上下文</h3> <p><strong><em>RequestContext</em></strong><br /> The request context contains all request relevant information. It is created at the beginning of the request and pushed to the <code class="highlighter-rouge">_request_ctx_stack</code> and removed at the end of it. It will create the URL adapter and request object for the WSGI environment provided</p> <p><strong><em>AppContext</em></strong>:<br /> The application context binds an application object implicitly to the current thread or greenlet.The application context is also implicitly created if a request context is created but the application is not on top of the individual application contex</p> <h3 id="helloworld">hello,world</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># coding: utf-8 from flask import Flask from flask import render_template from flask import request app = Flask(__name__) @app.route('/') def hello_world(): name = request.args.get('name', 'world') return 'Hello, ' + name @app.route('/hello/&lt;name&gt;') def hello(name=None): return render_template('hello.html', name=name) </code></pre></div></div> <p>flask run –host=0.0.0.0 <br /> or uwsgi -s /tmp/uwsgi.sock –module myapp –callable app –http 0.0.0.0:5000</p> <p>换成 django style 写 hello,world:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># index.py # coding: utf-8 from flask import Flask def create_app(): app = Flask(__name__, static_url_path='/static', static_folder="/webroot") import urls index = 0 for pattern in urls.urlpatterns: rule, view_func = pattern index += 1 app.add_url_rule(rule, endpoint=str(index), view_func=view_func) return app app = create_app() #urls.py import views urlpatterns = [ ('/', views.hello), ] # views.py def hello(): return 'Hello, World!' </code></pre></div></div> <p>更换成 views 类的方式:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#urls.py import views urlpatterns = [ ('/', views.MyView.as_view("index")), ] #views.py from flask import views class MyView(views.View): methods = ['GET'] def dispatch_request(self): return 'Hello World' </code></pre></div></div> <p>我们来看下 <code class="highlighter-rouge">flask.views</code> 这个类做了什么:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@classmethod def as_view(cls, name, *class_args, **class_kwargs): def view(*args, **kwargs): self = view.view_class(*class_args, **class_kwargs) return self.dispatch_request(*args, **kwargs) if cls.decorators: view.__name__ = name view.__module__ = cls.__module__ for decorator in cls.decorators: view = decorator(view) view.view_class = cls view.__name__ = name view.__doc__ = cls.__doc__ view.__module__ = cls.__module__ view.methods = cls.methods view.provide_automatic_options = cls.provide_automatic_options return view </code></pre></div></div> <h3 id="参考资料">参考资料</h3> <p><a href="https://blog.tonyseek.com/post/the-context-mechanism-of-flask/">https://blog.tonyseek.com/post/the-context-mechanism-of-flask/</a> <a href="http://werkzeug-docs-cn.readthedocs.io/zh_CN/latest/tutorial.html">http://werkzeug-docs-cn.readthedocs.io/zh_CN/latest/tutorial.html</a> <a href="https://jachinlin.github.io/2017/09/12/Werkzeug-URL-Routing/">https://jachinlin.github.io/2017/09/12/Werkzeug-URL-Routing/</a></p> minicap 一瞥 2018-06-06T00:00:00+08:00 /2018/06/06/about-minicap <h4 id="本地环境">本地环境</h4> <p>android sdk, android ndk-build tools(可以直接通过 android sdk manager安装)</p> <p>设置环境变量:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#ANDROID export ANDROID_HOME='/Users/guoliangyong/Library/Android/sdk' export PATH=$PATH:$ANDROID_HOME/platform-tools:$ANDROID_HOME/build-tools/26.0.0:$ANDROID_HOME/ndk-bundle </code></pre></div></div> <p>下载源码:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/openstf/minicap.git cd minicap &amp;&amp; git submodule init &amp;&amp; git submodule update ./run.sh autosize </code></pre></div></div> <h4 id="源码一瞥">源码一瞥</h4> 长时间没有写 blog 了 2018-05-15T00:00:00+08:00 /2018/05/15/long-time-no-write <p>已经一个多月没有写 blog 了,这么长时间里,我还是很平静的过着每一天。</p> <p>房租涨了 1千,北京的房子真是无可救药了(房租已经均价8千多了, 每平米均价88)。可怜。。。</p> <p>钱能解决的事情都不是事情,但「我没有钱啊。。。」</p> <p>技术上还是进步不大。。。</p> <p>「40岁了,还在写代码,是一种什么样的体验」? <br /> 中年危机,但是「做自己喜欢且擅长的事情」不是挺好?关键是,如果你聊到你「不可能擅长」或者做到「最好」,你会怎么去做?</p> <p>我不是为了变成 top 1% 的人,而是享受这个过程。在享受的过程中,慢慢地,变成了 top 1% ?</p> <p><a href="https://en.wikipedia.org/wiki/Quality_of_life">Quality of life</a> 的提升?</p> Gin, Golang HTTP web framework 2018-04-18T00:00:00+08:00 /2018/04/18/golang-gin-http-web-framework <p>近期几个 api server 使用 <a href="https://github.com/gin-gonic/gin">gin</a> 框架来实现。这里记录下 gin 的基本使用。 <br /> 来看 gin 的<code class="highlighter-rouge">hello,world</code>:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span> <span class="n">import</span> <span class="s2">"github.com/gin-gonic/gin"</span> <span class="n">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span> <span class="n">r</span> <span class="p">:=</span> <span class="n">gin</span><span class="p">.</span><span class="n">Default</span><span class="p">()</span> <span class="n">r</span><span class="p">.</span><span class="n">GET</span><span class="p">(</span><span class="s2">"/ping"</span><span class="p">,</span> <span class="n">func</span><span class="p">(</span><span class="n">c</span> <span class="p">*</span><span class="n">gin</span><span class="p">.</span><span class="n">Context</span><span class="p">)</span> <span class="p">{</span> <span class="n">c</span><span class="p">.</span><span class="n">JSON</span><span class="p">(</span><span class="m">200</span><span class="p">,</span> <span class="n">gin</span><span class="p">.</span><span class="n">H</span><span class="p">{</span> <span class="s2">"message"</span><span class="p">:</span> <span class="s2">"pong"</span><span class="p">,</span> <span class="p">})</span> <span class="p">})</span> <span class="n">r</span><span class="p">.</span><span class="nf">Run</span><span class="p">()</span> <span class="p">//</span> <span class="n">listen</span> <span class="k">and</span> <span class="n">serve</span> <span class="n">on</span> <span class="m">0.0.0.0</span><span class="p">:</span><span class="m">8080</span> <span class="p">}</span> </code></pre></div></div> <p>ok, 如果我们使用原生的 <a href="https://golang.org/doc/articles/wiki/">http 服务</a>怎么写呢?</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span> <span class="n">import</span> <span class="p">(</span> <span class="s2">"net/http"</span> <span class="p">)</span> <span class="n">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span> <span class="n">http</span><span class="p">.</span><span class="n">HandleFunc</span><span class="p">(</span><span class="s2">"/ping"</span><span class="p">,</span> <span class="n">func</span><span class="p">(</span><span class="n">w</span> <span class="n">http</span><span class="p">.</span><span class="n">ResponseWriter</span><span class="p">,</span> <span class="n">r</span> <span class="p">*</span><span class="n">http</span><span class="p">.</span><span class="n">Request</span><span class="p">)</span> <span class="p">{</span> <span class="n">w</span><span class="p">.</span><span class="n">Header</span><span class="p">().</span><span class="k">Set</span><span class="p">(</span><span class="s2">"Content-Type"</span><span class="p">,</span> <span class="s2">"application/json"</span><span class="p">)</span> <span class="n">w</span><span class="p">.</span><span class="n">Write</span><span class="p">([]</span><span class="n">byte</span><span class="p">(</span><span class="s2">"{</span><span class="se">\"</span><span class="s2">message</span><span class="se">\"</span><span class="s2">:</span><span class="se">\"</span><span class="s2">pong</span><span class="se">\"</span><span class="s2">}"</span><span class="p">))</span> <span class="p">})</span> <span class="n">http</span><span class="p">.</span><span class="n">ListenAndServe</span><span class="p">(</span><span class="s2">":9095"</span><span class="p">,</span> <span class="n">nil</span><span class="p">)</span> <span class="p">}</span> </code></pre></div></div> <p>看上去差不太多?再看 gin 提供的强大能力,把官方例子下载到本地: <br /> <code class="highlighter-rouge">curl https://raw.githubusercontent.com/gin-gonic/gin/master/examples/basic/main.go &gt; main.go</code></p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span> <span class="n">import</span> <span class="p">(</span> <span class="s2">"github.com/gin-gonic/gin"</span> <span class="p">)</span> <span class="n">var</span> <span class="n">DB</span> <span class="p">=</span> <span class="n">make</span><span class="p">(</span><span class="n">map</span><span class="p">[</span><span class="k">string</span><span class="p">]</span><span class="k">string</span><span class="p">)</span> <span class="n">func</span> <span class="n">setupRouter</span><span class="p">()</span> <span class="p">*</span><span class="n">gin</span><span class="p">.</span><span class="n">Engine</span> <span class="p">{</span> <span class="p">//</span> <span class="n">Disable</span> <span class="n">Console</span> <span class="n">Color</span> <span class="p">//</span> <span class="n">gin</span><span class="p">.</span><span class="n">DisableConsoleColor</span><span class="p">()</span> <span class="n">r</span> <span class="p">:=</span> <span class="n">gin</span><span class="p">.</span><span class="n">Default</span><span class="p">()</span> <span class="p">//</span> <span class="n">Get</span> <span class="err">请求</span> <span class="n">r</span><span class="p">.</span><span class="n">GET</span><span class="p">(</span><span class="s2">"/ping"</span><span class="p">,</span> <span class="n">func</span><span class="p">(</span><span class="n">c</span> <span class="p">*</span><span class="n">gin</span><span class="p">.</span><span class="n">Context</span><span class="p">)</span> <span class="p">{</span> <span class="n">c</span><span class="p">.</span><span class="k">String</span><span class="p">(</span><span class="m">200</span><span class="p">,</span> <span class="s2">"pong"</span><span class="p">)</span> <span class="p">//</span> <span class="err">返回字符串</span> <span class="p">})</span> <span class="p">//</span> <span class="n">Get</span> <span class="n">user</span> <span class="n">value</span> <span class="n">r</span><span class="p">.</span><span class="n">GET</span><span class="p">(</span><span class="s2">"/user/:name"</span><span class="p">,</span> <span class="n">func</span><span class="p">(</span><span class="n">c</span> <span class="p">*</span><span class="n">gin</span><span class="p">.</span><span class="n">Context</span><span class="p">)</span> <span class="p">{</span> <span class="n">user</span> <span class="p">:=</span> <span class="n">c</span><span class="p">.</span><span class="n">Params</span><span class="p">.</span><span class="n">ByName</span><span class="p">(</span><span class="s2">"name"</span><span class="p">)</span> <span class="p">//</span> <span class="err">获取参数</span> <span class="n">value</span><span class="p">,</span> <span class="n">ok</span> <span class="p">:=</span> <span class="n">DB</span><span class="p">[</span><span class="n">user</span><span class="p">]</span> <span class="k">if</span> <span class="n">ok</span> <span class="p">{</span> <span class="n">c</span><span class="p">.</span><span class="n">JSON</span><span class="p">(</span><span class="m">200</span><span class="p">,</span> <span class="n">gin</span><span class="p">.</span><span class="n">H</span><span class="p">{</span><span class="s2">"user"</span><span class="p">:</span> <span class="n">user</span><span class="p">,</span> <span class="s2">"value"</span><span class="p">:</span> <span class="n">value</span><span class="p">})</span> <span class="p">//</span> <span class="err">返回</span> <span class="n">JSON</span> <span class="err">数据</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="n">c</span><span class="p">.</span><span class="n">JSON</span><span class="p">(</span><span class="m">200</span><span class="p">,</span> <span class="n">gin</span><span class="p">.</span><span class="n">H</span><span class="p">{</span><span class="s2">"user"</span><span class="p">:</span> <span class="n">user</span><span class="p">,</span> <span class="s2">"status"</span><span class="p">:</span> <span class="s2">"no value"</span><span class="p">})</span> <span class="p">}</span> <span class="p">})</span> <span class="p">//</span> <span class="n">Authorized</span> <span class="n">group</span> <span class="p">(</span><span class="k">uses</span> <span class="n">gin</span><span class="p">.</span><span class="n">BasicAuth</span><span class="p">()</span> <span class="n">middleware</span><span class="p">)</span> <span class="p">//</span> <span class="n">Same</span> <span class="n">than</span><span class="p">:</span> <span class="p">//</span> <span class="n">authorized</span> <span class="p">:=</span> <span class="n">r</span><span class="p">.</span><span class="n">Group</span><span class="p">(</span><span class="s2">"/"</span><span class="p">)</span> <span class="p">//</span> <span class="n">authorized</span><span class="p">.</span><span class="n">Use</span><span class="p">(</span><span class="n">gin</span><span class="p">.</span><span class="n">BasicAuth</span><span class="p">(</span><span class="n">gin</span><span class="p">.</span><span class="n">Credentials</span><span class="p">{</span> <span class="p">//</span> <span class="s2">"foo"</span><span class="p">:</span> <span class="s2">"bar"</span><span class="p">,</span> <span class="p">//</span> <span class="s2">"manu"</span><span class="p">:</span> <span class="s2">"123"</span><span class="p">,</span> <span class="p">//}))</span> <span class="n">authorized</span> <span class="p">:=</span> <span class="n">r</span><span class="p">.</span><span class="n">Group</span><span class="p">(</span><span class="s2">"/"</span><span class="p">,</span> <span class="n">gin</span><span class="p">.</span><span class="n">BasicAuth</span><span class="p">(</span><span class="n">gin</span><span class="p">.</span><span class="n">Accounts</span><span class="p">{</span> <span class="s2">"foo"</span><span class="p">:</span> <span class="s2">"bar"</span><span class="p">,</span> <span class="p">//</span> <span class="n">user</span><span class="p">:</span><span class="n">foo</span> <span class="n">password</span><span class="p">:</span><span class="n">bar</span> <span class="s2">"manu"</span><span class="p">:</span> <span class="s2">"123"</span><span class="p">,</span> <span class="p">//</span> <span class="n">user</span><span class="p">:</span><span class="n">manu</span> <span class="n">password</span><span class="p">:</span><span class="m">123</span> <span class="p">}))</span> <span class="p">//</span> <span class="err">认证检查</span> <span class="n">authorized</span><span class="p">.</span><span class="n">POST</span><span class="p">(</span><span class="s2">"admin"</span><span class="p">,</span> <span class="n">func</span><span class="p">(</span><span class="n">c</span> <span class="p">*</span><span class="n">gin</span><span class="p">.</span><span class="n">Context</span><span class="p">)</span> <span class="p">{</span> <span class="n">user</span> <span class="p">:=</span> <span class="n">c</span><span class="p">.</span><span class="n">MustGet</span><span class="p">(</span><span class="n">gin</span><span class="p">.</span><span class="n">AuthUserKey</span><span class="p">).(</span><span class="k">string</span><span class="p">)</span> <span class="p">//</span> <span class="n">Parse</span> <span class="n">JSON</span> <span class="n">var</span> <span class="n">json</span> <span class="n">struct</span> <span class="p">{</span> <span class="n">Value</span> <span class="k">string</span> <span class="p">`</span><span class="n">json</span><span class="p">:</span><span class="s2">"value"</span> <span class="n">binding</span><span class="p">:</span><span class="s2">"required"</span><span class="p">`</span> <span class="p">}</span> <span class="k">if</span> <span class="n">c</span><span class="p">.</span><span class="n">Bind</span><span class="p">(&amp;</span><span class="n">json</span><span class="p">)</span> <span class="p">==</span> <span class="n">nil</span> <span class="p">{</span> <span class="n">DB</span><span class="p">[</span><span class="n">user</span><span class="p">]</span> <span class="p">=</span> <span class="n">json</span><span class="p">.</span><span class="n">Value</span> <span class="n">c</span><span class="p">.</span><span class="n">JSON</span><span class="p">(</span><span class="m">200</span><span class="p">,</span> <span class="n">gin</span><span class="p">.</span><span class="n">H</span><span class="p">{</span><span class="s2">"status"</span><span class="p">:</span> <span class="s2">"ok"</span><span class="p">})</span> <span class="p">}</span> <span class="p">})</span> <span class="n">return</span> <span class="n">r</span> <span class="p">}</span> <span class="n">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span> <span class="n">r</span> <span class="p">:=</span> <span class="n">setupRouter</span><span class="p">()</span> <span class="p">//</span> <span class="n">Listen</span> <span class="k">and</span> <span class="n">Server</span> <span class="k">in</span> <span class="m">0.0.0.0</span><span class="p">:</span><span class="m">8080</span> <span class="n">r</span><span class="p">.</span><span class="nf">Run</span><span class="p">(</span><span class="s2">":9095"</span><span class="p">)</span> <span class="p">}</span> </code></pre></div></div> <h3 id="初探">初探</h3> <p>看上面的例子,你几乎只需要定义好 path, handlefunc 即可:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>type HandlerFunc func(*Context) </code></pre></div></div> <p>gin 实现了 servehttp 的方法:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// Conforms to the http.Handler interface. func (engine *Engine) ServeHTTP(w http.ResponseWriter, req *http.Request) { c := engine.pool.Get().(*Context) c.writermem.reset(w) c.Request = req c.reset() engine.handleHTTPRequest(c) engine.pool.Put(c) } </code></pre></div></div> <p>详细的 path -&gt; handle 的逻辑这里不详细展开.</p> <h3 id="参数获取">参数获取</h3> <p>获取 path 中变量:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>router.GET("/user/:name", func(c *gin.Context) { name := c.Param("name") c.String(http.StatusOK, "Hello %s", name) }) </code></pre></div></div> <p>获取URL中参数:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> // The request responds to a url matching: /welcome?firstname=Jane&amp;lastname=Doe router.GET("/welcome", func(c *gin.Context) { firstname := c.DefaultQuery("firstname", "Guest") lastname := c.Query("lastname") // shortcut for c.Request.URL.Query().Get("lastname") c.String(http.StatusOK, "Hello %s %s", firstname, lastname) }) </code></pre></div></div> <p>获取 POST 请求参数:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>//POST /post?id=1234&amp;page=1 HTTP/1.1 //Content-Type: application/x-www-form-urlencoded // //name=manu&amp;message=this_is_great router.POST("/form_post", func(c *gin.Context) { message := c.PostForm("message") nick := c.DefaultPostForm("nick", "anonymous") c.JSON(200, gin.H{ "status": "posted", "message": message, "nick": nick, }) }) </code></pre></div></div> <p>获取上传文件:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>file, _ := c.FormFile("file") //or form, _ := c.MultipartForm() files := form.File["upload[]"] </code></pre></div></div> <h3 id="返回">返回</h3> <p>HTML :</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>router.LoadHTMLGlob("templates/*") c.HTML(http.StatusOK, "index.tmpl", gin.H{ "title": "Main website", }) </code></pre></div></div> <p>JSON 格式:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>c.JSON(http.StatusOK, gin.H{"message": "hey", "status": http.StatusOK}) var msg struct { Name string `json:"user"` Message string Number int } msg.Name = "Lena" msg.Message = "hey" msg.Number = 123 // Note that msg.Name becomes "user" in the JSON // Will output : {"user": "Lena", "Message": "hey", "Number": 123} c.JSON(http.StatusOK, msg) </code></pre></div></div> <p>XML:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>c.XML(http.StatusOK, gin.H{"message": "hey", "status": http.StatusOK}) </code></pre></div></div> <p>静态文件:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>router.Static("/assets", "./assets") </code></pre></div></div> <h3 id="状态码">状态码</h3> <p>重定向:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>c.Redirect(http.StatusMovedPermanently, "http://www.google.com/") </code></pre></div></div> <h3 id="使用中间件">使用中间件</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>func (engine *Engine) Use(middleware ...HandlerFunc) IRoutes { engine.RouterGroup.Use(middleware...) engine.rebuild404Handlers() engine.rebuild405Handlers() return engine } </code></pre></div></div> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>r.Use(gin.Recovery()) </code></pre></div></div> <h3 id="注意">注意</h3> <blockquote> <p>When starting new Goroutines inside a middleware or handler, you SHOULD NOT use the original context inside it, you have to use a read-only copy.</p> </blockquote> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>r.GET("/long_async", func(c *gin.Context) { // create copy to be used inside the goroutine cCp := c.Copy() go func() { // simulate a long task with time.Sleep(). 5 seconds time.Sleep(5 * time.Second) // note that you are using the copied context "cCp", IMPORTANT log.Println("Done! in path " + cCp.Request.URL.Path) }() }) </code></pre></div></div> <h3 id="多服务">多服务</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span> <span class="n">import</span> <span class="p">(</span> <span class="s2">"log"</span> <span class="s2">"net/http"</span> <span class="s2">"time"</span> <span class="s2">"github.com/gin-gonic/gin"</span> <span class="s2">"golang.org/x/sync/errgroup"</span> <span class="p">)</span> <span class="n">var</span> <span class="p">(</span> <span class="n">g</span> <span class="n">errgroup</span><span class="p">.</span><span class="n">Group</span> <span class="p">)</span> <span class="n">func</span> <span class="n">router01</span><span class="p">()</span> <span class="n">http</span><span class="p">.</span><span class="n">Handler</span> <span class="p">{</span> <span class="n">e</span> <span class="p">:=</span> <span class="n">gin</span><span class="p">.</span><span class="n">New</span><span class="p">()</span> <span class="n">e</span><span class="p">.</span><span class="n">Use</span><span class="p">(</span><span class="n">gin</span><span class="p">.</span><span class="n">Recovery</span><span class="p">())</span> <span class="n">e</span><span class="p">.</span><span class="n">GET</span><span class="p">(</span><span class="s2">"/"</span><span class="p">,</span> <span class="n">func</span><span class="p">(</span><span class="n">c</span> <span class="p">*</span><span class="n">gin</span><span class="p">.</span><span class="n">Context</span><span class="p">)</span> <span class="p">{</span> <span class="n">c</span><span class="p">.</span><span class="n">JSON</span><span class="p">(</span> <span class="n">http</span><span class="p">.</span><span class="n">StatusOK</span><span class="p">,</span> <span class="n">gin</span><span class="p">.</span><span class="n">H</span><span class="p">{</span> <span class="s2">"code"</span><span class="p">:</span> <span class="n">http</span><span class="p">.</span><span class="n">StatusOK</span><span class="p">,</span> <span class="s2">"error"</span><span class="p">:</span> <span class="s2">"Welcome server 01"</span><span class="p">,</span> <span class="p">},</span> <span class="p">)</span> <span class="p">})</span> <span class="n">return</span> <span class="n">e</span> <span class="p">}</span> <span class="n">func</span> <span class="n">router02</span><span class="p">()</span> <span class="n">http</span><span class="p">.</span><span class="n">Handler</span> <span class="p">{</span> <span class="n">e</span> <span class="p">:=</span> <span class="n">gin</span><span class="p">.</span><span class="n">New</span><span class="p">()</span> <span class="n">e</span><span class="p">.</span><span class="n">Use</span><span class="p">(</span><span class="n">gin</span><span class="p">.</span><span class="n">Recovery</span><span class="p">())</span> <span class="n">e</span><span class="p">.</span><span class="n">GET</span><span class="p">(</span><span class="s2">"/"</span><span class="p">,</span> <span class="n">func</span><span class="p">(</span><span class="n">c</span> <span class="p">*</span><span class="n">gin</span><span class="p">.</span><span class="n">Context</span><span class="p">)</span> <span class="p">{</span> <span class="n">c</span><span class="p">.</span><span class="n">JSON</span><span class="p">(</span> <span class="n">http</span><span class="p">.</span><span class="n">StatusOK</span><span class="p">,</span> <span class="n">gin</span><span class="p">.</span><span class="n">H</span><span class="p">{</span> <span class="s2">"code"</span><span class="p">:</span> <span class="n">http</span><span class="p">.</span><span class="n">StatusOK</span><span class="p">,</span> <span class="s2">"error"</span><span class="p">:</span> <span class="s2">"Welcome server 02"</span><span class="p">,</span> <span class="p">},</span> <span class="p">)</span> <span class="p">})</span> <span class="n">return</span> <span class="n">e</span> <span class="p">}</span> <span class="n">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span> <span class="n">server01</span> <span class="p">:=</span> <span class="p">&amp;</span><span class="n">http</span><span class="p">.</span><span class="n">Server</span><span class="p">{</span> <span class="n">Addr</span><span class="p">:</span> <span class="s2">":8080"</span><span class="p">,</span> <span class="n">Handler</span><span class="p">:</span> <span class="n">router01</span><span class="p">(),</span> <span class="n">ReadTimeout</span><span class="p">:</span> <span class="m">5</span> <span class="p">*</span> <span class="n">time</span><span class="p">.</span><span class="n">Second</span><span class="p">,</span> <span class="n">WriteTimeout</span><span class="p">:</span> <span class="m">10</span> <span class="p">*</span> <span class="n">time</span><span class="p">.</span><span class="n">Second</span><span class="p">,</span> <span class="p">}</span> <span class="n">server02</span> <span class="p">:=</span> <span class="p">&amp;</span><span class="n">http</span><span class="p">.</span><span class="n">Server</span><span class="p">{</span> <span class="n">Addr</span><span class="p">:</span> <span class="s2">":8081"</span><span class="p">,</span> <span class="n">Handler</span><span class="p">:</span> <span class="n">router02</span><span class="p">(),</span> <span class="n">ReadTimeout</span><span class="p">:</span> <span class="m">5</span> <span class="p">*</span> <span class="n">time</span><span class="p">.</span><span class="n">Second</span><span class="p">,</span> <span class="n">WriteTimeout</span><span class="p">:</span> <span class="m">10</span> <span class="p">*</span> <span class="n">time</span><span class="p">.</span><span class="n">Second</span><span class="p">,</span> <span class="p">}</span> <span class="n">g</span><span class="p">.</span><span class="n">Go</span><span class="p">(</span><span class="n">func</span><span class="p">()</span> <span class="n">error</span> <span class="p">{</span> <span class="n">return</span> <span class="n">server01</span><span class="p">.</span><span class="n">ListenAndServe</span><span class="p">()</span> <span class="p">})</span> <span class="n">g</span><span class="p">.</span><span class="n">Go</span><span class="p">(</span><span class="n">func</span><span class="p">()</span> <span class="n">error</span> <span class="p">{</span> <span class="n">return</span> <span class="n">server02</span><span class="p">.</span><span class="n">ListenAndServe</span><span class="p">()</span> <span class="p">})</span> <span class="k">if</span> <span class="n">err</span> <span class="p">:=</span> <span class="n">g</span><span class="p">.</span><span class="nf">Wait</span><span class="p">();</span> <span class="n">err</span> <span class="c1">!= nil { </span> <span class="nb">log</span><span class="p">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="p">}</span> </code></pre></div></div> <p>未完待续</p> Jmeter 介绍 2018-04-05T00:00:00+08:00 /2018/04/05/jmeter-usermanual <h3 id="前言">前言</h3> <p>在做压力测试前,先思考一下:</p> <ul> <li>预期平均用户数是多少?</li> <li>预期高峰用户数是多少?</li> </ul> <h3 id="概念">概念</h3> <h4 id="test-plan-测试计划">Test Plan (测试计划)</h4> <p>测试计划,即 Jmeter 的执行内容,完整的测试计划包括: Thread Groups(线程组), logic controllers(逻辑控制器), sample generating controllers(取样控制器), listeners(监听器), timers(定时器), assertions(断言), configuration elements(配置元素).</p> <p>一个测试计划,至少要包含: 一个线程组,一个或多个取样器。</p> <h4 id="thread-groups线程组">Thread Groups(线程组)</h4> <p>测试计划的入口。用于设置, 线程数,递增线程时间、总的运行次数与时间(持续时间)。</p> <p>eg:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Test Plan Thread Group - Once Only Controller * Login Request (an HTTP Request) - Load Search Page (HTTP Sampler) - Interleave Controller * Search "A" (HTTP Sampler) * Search "B" (HTTP Sampler) * HTTP default request (Configuration Element) - HTTP default request (Configuration Element) - Cookie Manager (Configuration Element) </code></pre></div></div> <p>上面的测试线程组执行一次 only once controller, 然后加载 search page, 交替执行 search a/b(按顺序交替选择),执行顺序是:</p> <ul> <li>Configuration elements</li> <li>Pre-Processors</li> <li>Timers</li> <li>Sampler</li> <li>Post-Processors (unless SampleResult is null)</li> <li>Assertions (unless SampleResult is null)</li> <li>Listeners (unless SampleResult is null)</li> </ul> <h4 id="controllers-控制器">controllers( 控制器)</h4> <p>JMeter 有两种控制器:Samplers and Logical Controllersa 。 Samplers: 请求方式。(http, ftp, tcp etc)<br /> Logical controllers: 发送请求的逻辑,loop/while/once only etc.</p> <h4 id="listeners监听器">Listeners(监听器)</h4> <p>监听器用来对 JMeter test cases 的执行进行展示、汇总、分析。View Results Tree, Graph Results listeners,Aggregate Report 等等。</p> <h4 id="timers定时器">Timers(定时器)</h4> <p>用于增加请求间的延时。</p> <h4 id="assertions-断言">assertions (断言)</h4> <p>用来测试、标记请求是否成功</p> <h4 id="pre-processorpost-processor">Pre-Processor/Post-Processor</h4> <p>用于请求的前后置处理。比如使用 HTTP URL Re-writing Modifier 来重置URL。</p> <h3 id="分布式部署"><a href="https://jmeter.apache.org/usermanual/remote-test.html">分布式部署</a></h3> <p>我们使用一台机器来进行说明(实际上,最好每台机器部署一个slave 节点)。</p> <p>启动 server, 端口号 9091/9092/9093:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SERVER_PORT=9091 /usr/local/Cellar/jmeter/3.2/libexec/bin/jmeter-server -Djava.rmi.server.hostname=localhost SERVER_PORT=9092 /usr/local/Cellar/jmeter/3.2/libexec/bin/jmeter-server -Djava.rmi.server.hostname=localhost SERVER_PORT=9093 /usr/local/Cellar/jmeter/3.2/libexec/bin/jmeter-server -Djava.rmi.server.hostname=localhost </code></pre></div></div> <p>使用 GUI 方式,启动 client or master,</p> <p>vi /usr/local/Cellar/jmeter/3.2/libexec/bin/jmeter.properties, 修改 remote_hosts:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>remote_hosts=127.0.0.1:9091,127.0.0.1:9092,127.0.0.1:9093 </code></pre></div></div> <p>使用非 GUI 方式启动:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jmeter -n -t ~/Downloads/URLRewritingExample.jmx -R127.0.0.1:9093,127.0.0.1:9091,127.0.0.1:9092 </code></pre></div></div> <h3 id="监控">监控</h3> <p>这里在 mac 下安装 influxdb 与 grafana 来配置<a href="https://jmeter.apache.org/usermanual/realtime-results.html">实时监控</a></p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>brew install influxdb brew services start influxdb </code></pre></div></div> <p>influx -precision rfc3339</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CREATE DATABASE grafana CREATE DATABASE jmeter SHOW DATABASES; </code></pre></div></div> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>brew install grafana brew services start grafana </code></pre></div></div> <p>打开: http://localhost:3000/ 登录设置 <br /> 添加 infuxdb 源。</p> <p>jmeter 测试时,添加 backend listeners:<code class="highlighter-rouge">http://127.0.0.1:8086/write?db=jmeter</code> 试试查看数据。可以在命令行中查看 influxdb 中的数据:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SHOW MEASUREMENTS; SHOW TAG KEYS FROM "jmeter"; select * from jmeter </code></pre></div></div> <p>维度:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>show field keys time application avg count countError endedT hit max maxAT meanAT min minAT pct90.0 pct95.0 pct99.0 startedT statut transaction </code></pre></div></div> <h3 id="最佳实践">最佳实践</h3> <h4 id="定义变量">定义变量</h4> <p>User Defined Variables, 定义变量:<br /> eg: name: resources_folder, value: /, desc: folder path</p> <p>Define CSV DATA:<br /> name: loginData <br /> Variable Name(comma-delimited): login, password <br /> <a href="http://jmeter.apache.org/usermanual/component_reference.html#CSV_Data_Set_Config">share mode</a>: All threads,所有线程组共享文件;Current thread group: 当前的线程组共享文件;Current thread: 线程共享,线程组内不共享。</p> <p>假如你有1万个用户的文件,想要每个线程的用户不同,则设置成Current thread group;如果可以相同,则可以设置成 Current thread.</p> <h4 id="减少资源使用">减少资源使用</h4> <ul> <li>使用非 GUI 模式: jmeter -n -t test.jmx -l test.jtl</li> <li>尽量少使用 Listeners, 不在压测是使用: “View Results Tree” or “View Results in Table”</li> <li>不实用 functional mode</li> <li>Use CSV output rather than XML</li> <li>只保存必须的数据</li> <li>尽量少使用断言</li> <li></li> </ul> <h3 id="提醒">提醒</h3> <p><code class="highlighter-rouge">Using GUI mode as described here should only be used when debugging your Test Plan. To run the real load test, use NON-GUI mode.</code></p> <p>压测千万别用 GUI 模式,一定要用 模式.</p> <h3 id="参考资料">参考资料</h3> <p><a href="https://www.ibm.com/developerworks/cn/opensource/os-pressiontest/">使用 JMeter 完成常用的压力测试</a> <br /> <a href="https://www.blazemeter.com/blog/how-to-use-grafana-to-monitor-jmeter-non-gui-results-part-2">How to Use Grafana to Monitor JMeter Non-GUI Results - Part 2</a></p> Ali Cloud Set Up 2018-04-03T00:00:00+08:00 /2018/04/03/alicloud-setup <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>useradd -m -d /home/myersguo -s /bin/bash myersguo passwd myersguo sudo usermod -a -G sudo myersguo sed -i "s/PasswordAuthentication yes/PasswordAuthentication no/g" /etc/ssh/sshd_config mkdir /home/myersguo/data /home/myersguo/app /home/myersguo/logs /home/myersguo/logs/applogs /home/myersguo/logs/php sudo apt-get update apt-get install net-tools apt-get install libpcre3-dev libssl-dev perl make build-essential curl </code></pre></div></div> <p>镜像地址:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># /etc/apt/sources.list deb http://mirrors.cloud.aliyuncs.com/debian/ stretch main contrib non-free deb http://mirrors.cloud.aliyuncs.com/debian/ stretch-updates main contrib non-free deb http://mirrors.cloud.aliyuncs.com/debian/ stretch-proposed-updates main non-free contrib deb http://mirrors.cloud.aliyuncs.com/debian/ stretch-backports main non-free contrib ## Major bug fix updates produced after the final release of the ## distribution. deb-src http://mirrors.cloud.aliyuncs.com/debian/ stretch-updates main contrib non-free deb-src http://mirrors.cloud.aliyuncs.com/debian/ stretch main contrib non-free deb-src http://mirrors.cloud.aliyuncs.com/debian/ stretch-proposed-updates main contrib non-free deb-src http://mirrors.cloud.aliyuncs.com/debian/ stretch-backports main contrib non-free deb http://mirrors.cloud.aliyuncs.com/debian-security/ stretch/updates main non-free contrib deb-src http://mirrors.cloud.aliyuncs.com/debian-security/ stretch/updates main non-free contrib </code></pre></div></div> <h3 id="mysql">mysql</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apt-get install -y mysql-server mysql-client mysql_secure_installation /etc/mysql/my.cnf /etc/systemd/system/mysql.service /etc/systemd/system/mysqld.service /etc/systemd/system/multi-user.target.wants/mariadb.service </code></pre></div></div> <h3 id="git">git</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt-get install git-core </code></pre></div></div> <h3 id="nginxopenresty">nginx/openresty</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apt-get install libpcre3-dev \ libssl-dev perl make build-essential curl PATH=/usr/local/openresty/nginx/sbin:$PATH apt-get install -y nginx-full lsof -i :80 </code></pre></div></div> <h3 id="setup-my-blog">setup my blog</h3> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt-get install ruby-full git clone https://github.com/myersguo/myersguo.github.com.git </code></pre></div></div> <p>参考<a href="https://help.github.com/articles/setting-up-your-github-pages-site-locally-with-jekyll/">例子</a>, 生成静态文件.启动 nginx 配置:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>error_log logs/error.log; events { worker_connections 1024; } http { server { listen 80; root /home/myersguo/app/myersguo.github.com/_site; include /usr/local/openresty/nginx/conf/mime.types; default_type application/octet-stream; index index.html index.htm; location / { } } } </code></pre></div></div> <p>启动: <code class="highlighter-rouge">nginx -p /home/myersguo -c conf/nginx.conf</code></p> About HyperLogLog 2018-03-12T00:00:00+08:00 /2018/03/12/about-HyperLogLog <h3 id="统计">统计</h3> <p>我们之前谈到过<a href="https://myersguo.github.io/2017/10/15/redis-bitmap-2.html">日活统计</a>的问题:</p> <p>假如直接用集合统计<code class="highlighter-rouge">sadd uv user_1</code>,那么要存储 1亿数据的用户数据,占用内存将达到<strong>1.02G</strong>(uid 按照11位存储,11<em>8</em>100000000)。如果采用 bitmap 的形式 <code class="highlighter-rouge">SETBIT uv 1 1</code>,那么存储1亿数据,内存将需要12M( 100000000 bit), 大大节省了空间。但是,如果 user_id 不稀疏,11 位的uid 将占用1.16G (10000000000) 空间。从之前的文章,我们知道在大数统计上,使用<strong>HyperLogLog</strong>算法来统计占用空间将大大缩短。我们来解开它的面纱。</p> <h3 id="基数估算">基数估算</h3> <p>上面我们提到,11 位的 uid 的 bit 存储大概占用1.16G,能表示 100亿的数据,但是我们通过的用户数都在10亿以内,我们仅仅使用10位bit就可以表示。那我们的为题就变成了,如果用10位bit来表示最大100亿的10以内的数据。</p> <p>显然,如果有一个 hash 函数,将100亿的数据,均匀的 hash 到10位 bit 内即可。</p> <p>1990年,一篇「A linear-time probabilistic counting algorithm for database applications」提出这种线性估算法。</p> <blockquote> <p>设有一哈希函数H,其哈希结果空间有m个值(最小值0,最大值m-1),并且哈希结果服从均匀分布。使用一个长度为m的bitmap,每个bit为一个桶,均初始化为0,设一个集合的基数为n,此集合所有元素通过H哈希到bitmap中,如果某一个元素被哈希到第k个比特并且第k个比特为0,则将其置为1。<a href="http://blog.codinglabs.org/articles/algorithms-for-cardinality-estimation-part-ii.html">link</a></p> </blockquote> <p><a href="https://gist.github.com/devdazed/3873524">这里</a> or <a href="https://github.com/addthis/stream-lib/blob/master/src/main/java/com/clearspring/analytics/stream/cardinality/LinearCounting.java">这里</a>有实现的例子。</p> <h3 id="hyperloglog">HyperLogLog</h3> <p>这篇<a href="http://content.research.neustar.biz/blog/hll.html">blog</a>演示了 hyperloglog 算法的过程:</p> <p>初始化一个长度为m(64)的数组,将待统计的集合分为m组, 每组的统计值放到对应的数组位置上。</p> <p>输入 value, 通过 hash 得到 hash value;<br /> hash value 的最后24位用二进制表示;<br /> 二进制bit位最后r位bit位,表示落在数组(桶)的位置index。<br /> 从第r+1位开始,第一个1出现的值,放到第index个位置。</p> <blockquote> <p>分桶平均的基本原理是将统计数据划分为mm个桶,每个桶分别统计各自的{k_{max}}k</p> </blockquote> <blockquote> <p>就是将哈希空间平均分成m份,每份称之为一个桶(bucket)。对于每一个元素,其哈希值的前k比特作为桶编号,其中2^k=m,而后L-k个比特作为真正用于基数估计的比特串。桶编号相同的元素被分配到同一个桶,在进行基数估计时,首先计算每个桶内元素最大的第一个“1”的位置,设为M[i],然后对这m个值取平均后再进行估计,即:</p> <p>n̂ =2^(1/m∑M[i]) 这相当于物理试验中经常使用的多次试验取平均的做法,可以有效消减因偶然性带来的误差。</p> </blockquote> <h3 id="参考资料">参考资料</h3> <p><a href="http://blog.codinglabs.org/articles/algorithms-for-cardinality-estimation-part-iii.html">http://blog.codinglabs.org/articles/algorithms-for-cardinality-estimation-part-iii.html</a> <br /> <a href="https://github.com/flier/pyfasthash">pyhasher</a><br /> <a href="http://www.rainybowe.com/blog/2017/07/13/%E7%A5%9E%E5%A5%87%E7%9A%84HyperLogLog%E7%AE%97%E6%B3%95/index.html">http://www.rainybowe.com/blog/2017/07/13/%E7%A5%9E%E5%A5%87%E7%9A%84HyperLogLog%E7%AE%97%E6%B3%95/index.html</a></p> python time and date 2018-03-09T00:00:00+08:00 /2018/03/09/python-time-date <h3 id="convert-str-to-timestamp">convert str to timestamp</h3> <p>如果在 PHP 中,直接使用:</p> <p><code class="highlighter-rouge">strtotime("2018-01-01")</code></p> <p>但在<code class="highlighter-rouge">python</code>中,这似乎成了一件头疼的事情。</p> <p>python 的<code class="highlighter-rouge">time</code> 是什么的?</p> <p><code class="highlighter-rouge">time</code> 这个 module 提供了函数管理时间值。有两种方式表示:</p> <ul> <li>seconds since the Epoch,in UTC</li> <li>tuple of 9 integers (year, month, day, hours, minutes, seconds, weekday, Julian day, DST flag)</li> </ul> <p>我们能用它来做什么呢?</p> <p>time.time() — 返回当前时间戳,可恨的是,它不支持参数,只能返回当前的,如果想要返回其他时间段的不可以</p> <p>time.localtime() — 返回当前的时间time.struct_time类型</p> <p>两种表示怎么转换呢?</p> <h4 id="str-to-timestamp">str to timestamp</h4> <p><strong>strptime-&gt;time.mktime</strong></p> <p>time.mktime(time.localtime())</p> <p>你如果需要将’2018-01-01’转换成时间戳,需要先用 tuple 表示:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>time.strptime("2018-01-01", "%Y-%m-%d") </code></pre></div></div> <p>然后转换成时间:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>time.mktime(time.strptime("2018-01-01")) </code></pre></div></div> <h3 id="timestamp-to-str">timestamp to str</h3> <p>先将时间戳转化成struct time:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>time.localtime(1520578947.458189) </code></pre></div></div> <p>再转成 string <strong>time.localtime-&gt;time.strftime</strong>:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># 格式化 time.strftime("%Y-%m-%d %H:%M:%S +0000", time.localtime(1520578947.458189)) # 直接返回 string time.asctime(time.localtime(1520578947.45889)) </code></pre></div></div> <p>可以看到,没有一个直接的 str2time 之类的方法,而是要有一个中间的 <code class="highlighter-rouge">struct_time</code> 阶段,WHY?</p> <h3 id="datetime">datetime</h3> <ul> <li>date(year, month, day) –&gt; date object</li> <li>datetime(year, month, day[, hour[, minute[, second[, microsecond[,tzinfo]]]]])</li> <li>tzinfo: bstract base class for time zone info objects</li> <li>time([hour[, minute[, second[, microsecond[, tzinfo]]]]]) –&gt; a time object</li> <li>timedelta: Difference between two datetime values</li> </ul> <h4 id="时区转换">时区转换</h4> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>from dateutil import tz src = datetime.datetime(2018,03,11) from_zone = tz.gettz('UTC') src_inzone = src.replace(tzinfo=from_zone) to_zone = tz.gettz('CST') src_inzone.astimezone(tz=to_zone) </code></pre></div></div> <h4 id="timestamp--string-datetime">timestamp string datetime</h4> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># datetime to string a = datetime.datetime(2018,01,01) a.strftime('%Y-%m-%d') # string to datetime datetime.datetime.strptime('20180101','%Y%m%d') # timestamp to datetime datetime.datetime.fromtimestamp(1520746754) # datetime to timestamp a.strftime('%s') # datetime to time tuple a.timetuple() # current datetime datetime.datetime.today(), datetime.datetime.now() </code></pre></div></div>